Creation of a shopping bot for pre-order madness

[Anasazi37]

5 hours ago, SpaceCowboy said:

Anyways, sharing my perspective.  It was a fun programming exercise, and it's not terribly complicated, but in the end, it is extremely fragile due to working on a black box that is subject to change and has conditions that are difficult to test for.

That is exactly why I stopped working on one last year after getting burned by the VF-1S Hikaru PO and many others prior to it. The black box nature of the sites, and their performance under extreme load, was on full display last night for the Roy PO and any bot designed to work with them would be extremely brittle/fragile. By the time you fix edge/corner cases that pop up in real time, the PO would be over. I've built large systems before, designed to handle a lot of traffic, and came to the same conclusion as you: it's not worth it. To do it "right" would be really expensive and would require a small team of people to monitor/update the bots during PO Madness. Unless you're trying resell Macross toys as a business, it doesn't make sense. And even then it doesn't make sense. You'd have to employ this strategy across multiple categories/genres/etc. to turn a reasonable profit.

Thanks for sharing your experience. I think it's a sobering example of what deploying a bot would actually entail.

[Big]

I understand.

Have you got an idea, how the scalpers are able to get so many items?

[Sanity is Optional]

Just now, Big said:

I understand.

Have you got an idea, how the scalpers are able to get so many items?

In some countries manpower and programmers are dirt cheap.

[HardlyNever]

Just adding to @SpaceCowboy, as I helped him with his bot and used it as well.

It basically confirms everything I said earlier; every individual retailer needs it own special bot, and they need to be constantly updated to have a chance at being successful.  We had to update ours like 3-4 times between inception and use (and we only focused on HLJ).

There is also the very real question of how "competitive" a homemade bot is. Finding an item and completing an order is one thing; having it do it faster than a human on PO madness night is another. Of course you could run multiple bots, but how many different accounts do you really want to have? 

The takeaway for me is that most of these things are actually being bought by humans, despite some people believing otherwise. Where bots are involved(and I do believe they are involved), they are made and maintained by "professional" scalpers that buy way more than Macross stuff, and do this on a scale your mere collector/enthusiast cannot keep up with.

Just my experience, a theory and my 2 cents.

 

Edited July 3, 2020 by HardlyNever

[crackpot]

3 minutes ago, HardlyNever said:

Just adding to @SpaceCowboy, as I helped him with his bot and used it as well.

It basically confirms everything I said earlier; every individual retailer needs it own special bot, and they need to be constantly updated to have a chance at being successful.  We had to update ours like 3-4 times between inception and use (and we only focused on HLJ).

There is also the very real question of how "competitive" a homemade bot is. Finding an item and completing an order is one thing; having it do it faster than a human on PO madness night is another. Of course you could run multiple bots, but how many different accounts do you really want to have? 

The takeaway for me is that most of these things are actually being bought by humans, despite some people believing otherwise. Were bots are involved(and I do believe they are involved), they are made and maintained by "professional" scalpers that buy way more than Macross stuff, and do this on a scale your mear collector/enthusiast cannot keep up with.

Just my experience, a theory and my 2 cents.

 

Wouldn't the bot not really work with Amiami and HLJ because you log in again always when checking out unless that's part of the programming and I don't know anything about coding?

[HardlyNever]

8 minutes ago, crackpot said:

Wouldn't the bot not really work with Amiami and HLJ because you log in again always when checking out unless that's part of the programming and I don't know anything about coding?

Logging in is one of the easier tasks to automate.

Overall speed is one of the biggest issues. Also, I'm 90% sure the major retailers intentionally change things (like IDs/labels) on their site to combat botting. So how often do you want to manually update your bot vs just ordering yourself is a legitimate question, for those of us that are only interested in a handful of products.

[SpaceCowboy]

Yeah, thanks to @HardlyNever for his assistance writing the bot. Should have mentioned that in the first post.  Lots of good tips. 

[Big]

@HardlyNeverThanks for the insight .

[Guyffon]

Come to think of it, its not unheard of that chinese scalpers can get quite a few humans to help them get stuff to scalp, examples being iphones and luxury brands. So I think you guys are right that some scalpers use many humans to help them make their purchase thus they have so many to sell.

[MacrossJunkie]

Here's something I've been wondering about. A good majority of North American sites I go to have some form of captcha, like type the characters you see in the box or click all the boxes that have X. However, none of the Japanese sites (that I've gone to anyway) implement this. Wouldn't that stop the automated bot scalpers or at least reduce them if they used some sort of anti-botting captcha scheme?

Edited July 18, 2020 by MacrossJunkie

[Shizuka the Cat]

On 7/3/2020 at 3:13 PM, Big said:

I understand.

Have you got an idea, how the scalpers are able to get so many items?

One approach is to use Selenium, a tool that can be used with, such programming languages as Python or Java to automate web navigation.

https://towardsdatascience.com/controlling-the-web-with-python-6fceb22c5f08 documents the use of Python + Selenium to automate homework assignment submission to a school's Canvas web page.  A similar approach could be applied for handling online shopping.

[seti88]

1 hour ago, MacrossJunkie said:

Here's something I've been wondering about. A good majority of North American sites I go to have some form of captcha, like type the characters you see in the box or click all the boxes that have X. However, none of the Japanese sites (that I've gone to anyway) implement this. Wouldn't that stop the automated bot scalpers or at least reduce them if they used some sort of anti-botting captcha scheme?

The ultimate anti-bot scheme is PAY UPFRONT. 

[MacrossJunkie]

4 minutes ago, seti88 said:

The ultimate anti-bot scheme is PAY UPFRONT. 

Well, a lot of stores do that and it still doesn't help...

They buy up the stock at the normal price and then sell it at a marked up value later.

[seti88]

3 minutes ago, MacrossJunkie said:

Well, a lot of stores do that and it still doesn't help...

They buy up the stock at the normal price and then sell it at a marked up value later.

I would like to see them try with high priced items in bulk. You gotta make sure you got the capital to pull the trigger. 

And also it would help to have the listings up longer as some need validations on cards etc.

Over time, it will be a drain on their resources.

Edited July 18, 2020 by seti88

[Anasazi37]

9 hours ago, Shizuka the Cat said:

One approach is to use Selenium, a tool that can be used with, such programming languages as Python or Java to automate web navigation.

https://towardsdatascience.com/controlling-the-web-with-python-6fceb22c5f08 documents the use of Python + Selenium to automate homework assignment submission to a school's Canvas web page.  A similar approach could be applied for handling online shopping.

Look no further than sneaker bots for insight into what works, and what doesn't, when it comes to building something like this for rare collectibles. Selenium and similar automation tools are mentioned in several articles as a good place to start, but successful bots tend to be a lot more complex. Sneaker collecting is big business, so folks have put a lot of work into their bots. It's quite the arms race between them and the vendors, so it's creating this weird evolutionary environment where a bot is deployed, then it's defeated, then it comes back stronger/faster/smarter, is defeated again, and so on. Pretty soon we'll have Skynet, but at least the Terminators will be wearing really nice shoes....

https://www.highsnobiety.com/p/sneaker-bots-2019/

https://www.highsnobiety.com/p/sneaker-bots-how-do-they-work/

https://medium.com/@ayinope/how-sneaker-bots-actually-work-a016fd419646

[DYRL VF-1S]

[HardlyNever]

15 hours ago, Shizuka the Cat said:

One approach is to use Selenium, a tool that can be used with, such programming languages as Python or Java to automate web navigation.

https://towardsdatascience.com/controlling-the-web-with-python-6fceb22c5f08 documents the use of Python + Selenium to automate homework assignment submission to a school's Canvas web page.  A similar approach could be applied for handling online shopping.

This is what the bot @SpaceCowboy made uses.  It works (it can successfully complete a transaction), it just isn't all that fast, and it is "fragile."  

Just to reiterate:  A bot is possible.  A fast bot is possible.  I'm fairly confident there are scalpers using them.  Making (and maintaining) one is not easy, but if you were doing this on a large scale (or were a group of people doing this on a large scale), the development and maintenance time would be worth it, especially if you live in a part of the world where your individual labor per hour is on the lower end.  Especially if you were doing this to say... scalp a lot of high demand/low supply products.

I'm not trying to discourage anyone from trying this, I was just weighing in with my experience. 

As far as what vendors can do to combat them (more than they already do, and I think they do some things (see above)), implementing a captcha of some kind would definitely help, but the push back from customers who are not aware of the situation would definitely be a headache.  If they were to ask me, I would tell them to implement it on the most scalped products (mainly the Bandai stuff) on a short term, when they go up for pre-order.  But there are still X number of customers who would still not understand why it is there, and how it (potentially) helps them, and they would definitely complain.  So I can understand why a lot of these vendors don't even bother.  Also, as I said before, I still believe most of the buyers are actually humans, so on our end, our experience wouldn't be much different (we'd just have to fill out a captcha before being told the product is sold out).

[seti88]

Watching all these shopping bot talk..reminds me of the scene with kaneda and tetsuo in akira. When kaneda realises he cant hurt tetsuo with his laser rifle anymore, kaneda rolls up his sleeve and shouts to tetsuo...cmon fight like a man!!(from my old English dub vhs back in the day)

All I have is my F5...hahaha...

Edited July 19, 2020 by seti88

[MacrossJunkie]

While looking around, in regards to feature set, this bot software looks pretty good, although it's $101 just for the standard version alone.

https://www.botchief.com/

Seems to be written in C++, claims it doesn't require programming knowledge, can use chrome as an editor, and apparently can even solve captchas too.

I'll just stick with the old F5/ web monitor method though... for now at least. Also I don't really trust this sort of software unless I can see the source code.

Edited July 19, 2020 by MacrossJunkie

[Anasazi37]

6 hours ago, MacrossJunkie said:

Also I don't really trust this sort of software unless I can see the source code.

Exactly. There's really no excuse to be doing closed-source development for something like this and C++ seems like an odd choice. That's like using a Formula One car for grocery shopping when a souped-up minivan is available. Modern web tools are built in higher-level languages like JavaScript/Python/Go/etc. for good reason.

[Guyffon]

On 7/18/2020 at 10:19 AM, MacrossJunkie said:

Here's something I've been wondering about. A good majority of North American sites I go to have some form of captcha, like type the characters you see in the box or click all the boxes that have X. However, none of the Japanese sites (that I've gone to anyway) implement this. Wouldn't that stop the automated bot scalpers or at least reduce them if they used some sort of anti-botting captcha scheme?

This won’t work if the scalpers use many humans to make the orders

Edited July 19, 2020 by Guyffon

[vladykins]

2 hours ago, Guyffon said:

This won’t work if the scalpers use many humans to make the orders

It will if they are using those "Pick out the street lights in the picture" captchas and they are as terrible as I am at deciding "what constitutes a streetlight?"

[SpaceCowboy]

I work in ai, and I'm half convinced those "choose all squares with thing x" are getting humans to label training data as much as they are seeing if you are human. 

[vladykins]

Then I apologize for setting your AI back a few decades with my selections.

[MacrossJunkie]

That's entirely possible that we're helping the machine learning algorithms with our selections. 

I too fail them often. So I guess I'm not human. But seriously, I don't know if when selecting all the squares that contain a traffic light, whether the yellow pixel from the traffic light housing that's sticking out into the next square counts or not. 

[Anasazi37]

3 hours ago, SpaceCowboy said:

I work in ai, and I'm half convinced those "choose all squares with thing x" are getting humans to label training data as much as they are seeing if you are human. 

I work in AI as well. That is absolutely the primary goal of those applications. Google freely admits that's why they use pictures of street addresses and other transportation infrastructure in those puzzles. Your inputs are used to refine the AI that underpins Google Street View.

[Radioguy]

Has anyone continued in this vein? I sure would love such a bot for Mandarake, and suspect one exists in Japan somewhere considering how fast some things sell.

[Anasazi37]

21 hours ago, Radioguy said:

Has anyone continued in this vein? I sure would love such a bot for Mandarake, and suspect one exists in Japan somewhere considering how fast some things sell.

I know that some of the proxy services in Japan use them for highly competitive preorders. Mandarake's site is simple enough that you could probably build something useful, but you could end up automatically buying a poor quality used item unless you add some natural language processing filters to catch that. Mandarake tends to use standard descriptions for item condition, so that helps. 

[Radioguy]

1 hour ago, Anasazi37 said:

I know that some of the proxy services in Japan use them for highly competitive preorders. Mandarake's site is simple enough that you could probably build something useful, but you could end up automatically buying a poor quality used item unless you add some natural language processing filters to catch that. Mandarake tends to use standard descriptions for item condition, so that helps. 

I could understand without keyword recognition one could end up with a lemon, but having recognition for words like "Unopened" or "Main item only" or just "Damaged" would seem easy to code for the description field. Still, I think many would roll the dice knowing it may not be perfect when setting a max price to buy that's not so high.

Yes, I'd be running it right now for a couple items I think I could only find there. At least at a price I can afford, anyhow.

I hope someone is still working on such an app, or can tell me where one exists, that would at least work with Mandarake.

[Sandman]

Though not good for preorders, I miss figinstock,com. Loved getting up every morning and checking my emails for that figinstock email listing all the item from my search criteria.