Creation of a shopping bot for pre-order madness

[crackpot]

2 hours ago, SpaceCowboy said:

or try to go to one of these sites and say "look we have 20-30 people that would order from you if you could gaurantee a preorder without the madness" and see if they would play ball.  That sounds more interesting to me as well.

 

The retailers aren't having problems selling all the Macross pre-order units they have so they don't really have an incentive to agree to this offer. Unless, the people agree to buy other stuff that aren't flying off their shelves. 

[Shizuka the Cat]

I should point out the following issues involved with developing a pre-ordering bot:

1.) If the bot is frequently querying an online store to determine when the item is available to pre-order, such activity will contribute to the site's showed performance.

2.) Proper development and testing of the bot would likely require doing an actual purchase from each online store you want the bot to support, in order to ensure that the bot understands each store's item selection and checkout workflow.  Admittedly, while one would probably have to go thru a complete check-out process from each store as part of the bot development, I suppose in most cases the developer could cancel their submitted testing pre-orders or purchases afterwards.

3.) Depending on the store, an item's pre-order page might not be immediately found in a store's search results. CD Japan comes to mind as an example. As such, a bot may require manual entry of an item's pre-order page... or even entry of every store's respective pre-order page... if store search functions lag behind in updating their listings.

4.) Some stores change how their websites work from time to time. As such the bot would need to be maintained to ensure store support.

My 2 bits, as a computer science student.

 

 

[SpaceCowboy]

Quote

The retailers aren't having problems selling all the Macross pre-order units they have so they don't really have an incentive to agree to this offer. Unless, the people agree to buy other stuff that aren't flying off their shelves. 

True.  The real problem is Bandai.  If they would up their production runs or simply have multiple windows to guage true demand, I don't think the problem would be nearly as bad.  I've said it before, feels like they are leaving a ton of money on the table.

[WaferMouse]

I've already contributed to this topic and in general I just think the bot is a bad idea, so I'll just leave my advice based on my 100% successful experience from PO nights:

- Book the day off if necessary.

- Check that your account details are correct the day before.

- Set your alarm with enough time to pee and login. Adjust your sleeping schedule over the preceding days, if necessary and possible. Prepare light clothing before sleeping.

- Use a desktop/laptop with a mouse attached. Absolutely do not use a phone.

- Use a landline internet connection, absolutely do not use cell data.

- Close all tabs and windows. You can always temporarily bookmark them in bulk if necessary.

- Disable all your plugins/extensions.

- Open one tab for each site you plan to try, and the relevant MW thread. Login beforehand.

- Use ctrl-tab to switch between sites and F5 to refresh.

- Be calm and ready to strike.

- If it's an absolute must-have piece for you, be prepared to swallow your pride and go with a site with a higher price or shipping cost. The scalpers are in it for the profit.

If you've followed all these steps, congratulations, YOU are the bot now, and you saved $5.

Edited November 14, 2019 by WaferMouse

[WaferMouse]

Oops, double post.

Edited November 14, 2019 by WaferMouse

[crackpot]

1 hour ago, SpaceCowboy said:

True.  The real problem is Bandai.  If they would up their production runs or simply have multiple windows to guage true demand, I don't think the problem would be nearly as bad.  I've said it before, feels like they are leaving a ton of money on the table.

Or take the Hot Toys/ Sideshow route and leave the PO open even just for 2 days to get all the orders in. 

[Slave IV]

Would everyone get their own bot for that price? Not to be pessimistic but based on all the feedback and other issues I can think of, they probably all lead to the reason why this hasn't been done yet. If the only catch is $5 and I get a bot that delivers preordered toys to me with no additional cost, I'm in!

[sqidd]

2 hours ago, WaferMouse said:

I've already contributed to this topic and in general I just think the bot is a bad idea, so I'll just leave my advice based on my 100% successful experience from PO nights:

- Book the day off if necessary.

- Check that your account details are correct the day before.

- Set your alarm with enough time to pee and login. Adjust your sleeping schedule over the preceding days, if necessary and possible. Prepare light clothing before sleeping.

- Use a desktop/laptop with a mouse attached. Absolutely do not use a phone.

- Use a landline internet connection, absolutely do not use cell data.

- Close all tabs and windows. You can always temporarily bookmark them in bulk if necessary.

- Disable all your plugins/extensions.

- Open one tab for each site you plan to try, and the relevant MW thread. Login beforehand.

- Use ctrl-tab to switch between sites and F5 to refresh.

- Be calm and ready to strike.

- If it's an absolute must-have piece for you, be prepared to swallow your pride and go with a site with a higher price or shipping cost. The scalpers are in it for the profit.

If you've followed all these steps, congratulations, YOU are the bot now, and you saved $5.

If I give you $10 will you do that for me? That sounds like a lot of work!

[spacemanoeuvres]

8 hours ago, sqidd said:

If I give you $10 will you do that for me? That sounds like a lot of work!

Lol 

[oshanmacross]

20 hours ago, WaferMouse said:

 

-  Prepare light clothing before sleeping.

 

Not even gonna ask....

[Sildani]

Not to brag, but I snagged my VF-31C and VF-1J preorders while lying in bed with my iPhone on Wi-Fi at Oh-Dark-Thirty. I wouldn’t necessarily recommend it, but thought I’d contribute anecdotal evidence that, yes, Virginia, miracles can happen. 

[Sanity is Optional]

The 31C wasn't too terribly hard to get, and the 1J was surprisingly easy (managed to get 3 of those!).

The Hikaru 1S was far worse than either.

[Guyffon]

If this is a possiblilty, do you guys only service those in the US of A or anyone on this board has access to it? I am from Singapore and I won’t mind $5 for a peace of mind, that said, the only other stuff on my radar now are the yf-21 and the hmr tomohawk, not interested in the roy 1s

[theraiden]

2 hours ago, Guyffon said:

If this is a possiblilty, do you guys only service those in the US of A or anyone on this board has access to it? I am from Singapore and I won’t mind $5 for a peace of mind, that said, the only other stuff on my radar now are the yf-21 and the hmr tomohawk, not interested in the roy 1s

If this would happen, and that's a big IF, anyone who would chip in (and we have no idea how much the specific amount would be since we haven't got a price quote yet and don't know exactly how many would chip in) we would give everyone access that chipped in and could show proof they are a Macross collector (to weed out most of the scalpers)

[Anasazi37]

On 11/13/2019 at 10:24 AM, HardlyNever said:

I understand the frustration of the situation, but personally, I don't think us botting is going to solve much of anything.  The only reason I'm considering developing one personally, is so that I don't have to get up at 3AM (my time) to do this crap manually.  That's my real incentive there.  I don't think it would improve my odds of success that much.

This is exactly my situation. I've been looking into personally developing a bot for several months now, off and on, because the few times I've participated in middle-of-the-night PO Madness over the past two years it had a substantial negative impact on my life for a day or two afterwards and I simply can't afford to let that happen. For example, Hikaru 1S PO Madness hit while I was on a business trip, so there I was, hunched over my laptop on a crappy hotel wireless connection at 1:30am, frantically hitting Ctrl+Tab and F5 across many sites, trying to snag one. I couldn't. And I was wrecked for my meetings later in the day (lots of coffee got me through). It was the last straw for me, so my current strategy is to do...nothing. I wake up early enough that if I can't find the item on PO day--even at NY with their infamous wallet-demolishing markup--I just put it on my list of things to grab after release because I'm patient and eventually one will show up on Mandarake, Jungle, or the For Sale thread here. Having a personal bot running during PO Madness would allow me to sleep and maybe grab one, but if it doesn't work, I'm right where I was, waiting to grab one later, which is fine (but generally much more expensive). However, developing a successful bot, tuned to one or more sites, will be a lot of work and, as @Shizuka the Cat says, it will require constant upkeep as the sites improve their countermeasures. It's an arms race, really, which is why I'm still on the fence about it. The sneaker world has been dealing with this issue for years and there are lots of articles and blog posts about sneakerbots (pros and cons). It's worth reading those before deciding one way or the other.

 

On 11/13/2019 at 10:24 AM, HardlyNever said:

In my opinion, if we were going to act collectively in some fashion, trying to change the behavior of one or more of the big retailers would be our best bet.  We have some people here that drop some pretty serious cash at some of these vendors.  Collectively, we might represent enough buying power to influence something like AmiAmi, HLJ, etc. to change their pre-order practice, at least when it comes to Bandai DX Macross products.  But that would require us acting collectively, not individually (and selfishly) to get preferential treatment for ourselves.

I've been having a conversation with Scott Hards (founder and CEO of HLJ) about this for a year and a half now. I reached out to him after I wasn't able to get a YF-19 Full Set Pack through his site, which was the latest in a long string of failures stretching all the way back to the original release of the VF-31J in 2016. I had stepped away from Macross collecting for about ten years and when I re-engaged, I found a completely different sales landscape filled with ruthless frontrunners and scalpers on one end, Bandai's low production numbers and indifference on the other, and stores stuck in the middle taking most/all of the heat because they are directly customer-facing. I remembered being able to jump onto HLJ, easily find what I want, and be done. Not anymore, and what we have to do now is taking all of the joy out of collecting. I'd been a customer of HLJ for a long time, and I guess you could say that I've spent a lot of money there over the years, but more importantly they do pride themselves on customer service, so I thought they might listen to me if I started asking questions about how the process could be more fair to collectors like us. So, I fired off a note to Scott directly, indicating that I was part of a larger group of passionate collectors, and he replied, kicking off a conversation that's still going. I can tell you that HLJ is *very much* aware of our collective frustration and they are looking at ways to help, but they are also squarely pointing the finger at Bandai, who only gives them a limited amount of stock. I threw a bunch of ideas at him, including the option to do a phased PO across multiple time zones, i.e., release a certain amount of stock one hour, then more the next, and so on, so that their global customer base has a fighting chance to get an item during local business hours--a process that could be easily automated. Right now, if you're in the Americas, you're at a distinct disadvantage despite the fact that we're supposed to be one of HLJ's key target audiences, and I made sure to point that out. He said that their team would add my ideas to the pool of options they are currently considering, but that whatever they decide to do, the biggest problem is limited stock and there will be lots of unhappy customers unless/until Bandai increases their production numbers. Since I have this conversation going with him, I'd be happy to pass along a curated set of comments and ideas. If we can get one of the big stores to change how they handle POs, I suspect that the other big ones will follow suit.

[Slave IV]

This is all good info. I just want to point out that while the limited stock is Bandai's fault, they probably have a really good reason for it and that being profit margins. We can already see that Macross products are extremely niche and then niche within the niche. It simply would make no sense to any large corporation to dedicate too much resource to the brand. So in the end, it's our fault for not buying more and influencing the collectors out there to buy more Macross. There is no shortage issue with Arcadia because they are a small outfit that depends and caters to Macross so they make everything pretty much to order so everyone gets a good shot to get what they want initially.

[Anasazi37]

2 hours ago, Slave IV said:

This is all good info. I just want to point out that while the limited stock is Bandai's fault, they probably have a really good reason for it and that being profit margins. We can already see that Macross products are extremely niche and then niche within the niche. It simply would make no sense to any large corporation to dedicate too much resource to the brand. So in the end, it's our fault for not buying more and influencing the collectors out there to buy more Macross. There is no shortage issue with Arcadia because they are a small outfit that depends and caters to Macross so they make everything pretty much to order so everyone gets a good shot to get what they want initially.

Especially when Bandai is focused on Japan. They likely do their production math with the domestic otaku market in mind and that crowd probably isn't very large. There are several threads in the Toys section that have gone round and round on the issue of whether or not Bandai cares about Macross collectors outside of Japan, which then usually leads to international copyright discussions, etc. I don't want to inject any of that here, just wanted to reinforce your point.

[Slave IV]

53 minutes ago, Anasazi37 said:

Especially when Bandai is focused on Japan. They likely do their production math with the domestic otaku market in mind and that crowd probably isn't very large. There are several threads in the Toys section that have gone round and round on the issue of whether or not Bandai cares about Macross collectors outside of Japan, which then usually leads to international copyright discussions, etc. I don't want to inject any of that here, just wanted to reinforce your point.

Yep. It's not worth the trouble to even consider marketing and producing Macross outside of Japan due to HG. Bottom line, preorder madness will ensue as long as Macross is as niche as it is and Bandai continues to produce it. No worries with Arcadia and other boutique shops but then people whine about the price. It's all pretty funny to me. I enjoy whatever I can get and if there is preorder madness, it's all a part of the game.

[theraiden]

22 hours ago, Anasazi37 said:

I've been having a conversation with Scott Hards (founder and CEO of HLJ) about this for a year and a half now. I reached out to him after I wasn't able to get a YF-19 Full Set Pack through his site, which was the latest in a long string of failures stretching all the way back to the original release of the VF-31J in 2016. I had stepped away from Macross collecting for about ten years and when I re-engaged, I found a completely different sales landscape filled with ruthless frontrunners and scalpers on one end, Bandai's low production numbers and indifference on the other, and stores stuck in the middle taking most/all of the heat because they are directly customer-facing. I remembered being able to jump onto HLJ, easily find what I want, and be done. Not anymore, and what we have to do now is taking all of the joy out of collecting. I'd been a customer of HLJ for a long time, and I guess you could say that I've spent a lot of money there over the years, but more importantly they do pride themselves on customer service, so I thought they might listen to me if I started asking questions about how the process could be more fair to collectors like us. So, I fired off a note to Scott directly, indicating that I was part of a larger group of passionate collectors, and he replied, kicking off a conversation that's still going. I can tell you that HLJ is *very much* aware of our collective frustration and they are looking at ways to help, but they are also squarely pointing the finger at Bandai, who only gives them a limited amount of stock. I threw a bunch of ideas at him, including the option to do a phased PO across multiple time zones, i.e., release a certain amount of stock one hour, then more the next, and so on, so that their global customer base has a fighting chance to get an item during local business hours--a process that could be easily automated. Right now, if you're in the Americas, you're at a distinct disadvantage despite the fact that we're supposed to be one of HLJ's key target audiences, and I made sure to point that out. He said that their team would add my ideas to the pool of options they are currently considering, but that whatever they decide to do, the biggest problem is limited stock and there will be lots of unhappy customers unless/until Bandai increases their production numbers. Since I have this conversation going with him, I'd be happy to pass along a curated set of comments and ideas. If we can get one of the big stores to change how they handle POs, I suspect that the other big ones will follow suit.

This is great! I wonder if Bandai would produce more for western fanbase if they *knew* beforehand how many units are really going to sell.
Like if HLJ and other stores would do deposit pre-orders with more lead time than the usual process as it is now.
That way Bandai would have a much clearer idea how many units they would actually sell and maybe produce more since non-refund deposit orders are much more binding.

[crackpot]

I like what @Anasazi37said about HLJ. If they are willing to do the phased PO time, that can be helpful for those in the EST to PST zones (well PST isn't that bad to be honest). 

My thinking here is if Big West has been winning back their copyright on other regions of the globe, shouldn't that pave the way for Bandai to increase production numbers and export them to these areas? With regular release Metal Build and SOC/DX super robot stuff, they're not that hard to get. I don't even bother competing on JP PO night. I wait for the domestic PO. If I miss that, guess what? There are some decent prices to be found from EU retailers and sometimes their PO goes up earlier than US POs.

[seti88]

29 minutes ago, theraiden said:

This is great! I wonder if Bandai would produce more for western fanbase if they *knew* beforehand how many units are really going to sell.
Like if HLJ and other stores would do deposit pre-orders with more lead time than the usual process as it is now.
That way Bandai would have a much clearer idea how many units they would actually sell and maybe produce more since non-refund deposit orders are much more binding.

I suspect bandai doesnt have an unlimited capacity to produce, even if they knew in advance how many pre-orders there are.
So, I have come to accept the fact that there will be shortages.

 

 

[wm cheng]

That's great

Anasazi37

that you have a dialog open with HLJ (my personal favourite to deal with even if its not the cheapest!).  I will be very interested in what you guys have to say.

[SpaceCowboy]

I'm not sure I buy the whole "poor Bandai, they can barely make a profit at the numbers they are producing and any more would be a loss."  We all loved getting cheap yamatos on sale months after release due to being shelf warmers - that is a very clear case of over production.  But most of what Bandai is selling sells out within minutes.  Not even hours or days.  It would be really interesting to know the minimum increments they could go up.  Maybe they produce 5000 units and the next smallest amount they could produce would be 10,000 and really they have demand for 7000.  That sucks, but that's the only situation I can really understand it not being profitable for them. 

I also don't buy that "well, they don't care about the US market", because if resellers in Japan buy it for resale in the US, Bandai doesn't really care, because the  first point of purchase was from Japan, so to them, it is just stock that was bought in Japan.

I think more likely is that Macross makes up such a small percentage of their revenue stream that if they leave a bit of money on the table, they don't really care as other properties such as Gundam, DragonBall, etc, dwarf Macross and it makes more sense to concentrate on them.

But they really do suck all the enthusiasm out of collecting these multi-hundred dollar, fragile, display pieces.

[SpaceCowboy]

Going back to the topic of making a bot...my guess is that the amount each person would have to pitch in for a bot would be as much if not more than just paying the mark up that some stores and scalpers already put on the items.  At least the first time.

I do think a decently trivial bot might be possible with something like the python Selenium library.

[Slave IV]

Hey @Anasazi37 is HLJ aware that their “new, improved” website is a pile of $#!+

They went from one of my top 2 foot shops to being a complete waste of time and worthless. I haven’t bought a single thing from them on preorder since and maybe only one or two sale items.

Maybe if they just fixed their POS site, that would help enough. Lol. 

[Guyffon]

Just my 2 cents worth, if you want absolute advantage, your bot must be on a server in japan, if its elsewhere, it might still be too slow to get the purchases you need, if the scalpers are using bots too, every milisec is important. Also from my experience, japan’s internet speeds are blistering, that alone will give the bot base on a japanese server an advantage.

[Anasazi37]

37 minutes ago, Guyffon said:

Just my 2 cents worth, if you want absolute advantage, your bot must be on a server in japan, if its elsewhere, it might still be too slow to get the purchases you need, if the scalpers are using bots too, every milisec is important. Also from my experience, japan’s internet speeds are blistering, that alone will give the bot base on a japanese server an advantage.

This is starting to sound more and more like high speed trading, a.k.a., frontrunning

[Anasazi37]

On 11/17/2019 at 12:03 PM, Slave IV said:

Hey @Anasazi37 is HLJ aware that their “new, improved” website is a pile of $#!+

I haven't hit them too hard about the new site, but that's next on my list. Some aspects of it are fine, but the big one impacting us is that crazy-long delay as they query their database for item availability, during which you can't do anything but wait. On PO Madness nights, it's the kiss of death. I have no idea why that is so excruciatingly slow. It's not like they're sequencing DNA, looking for exoplanets, or creating Skynet. It's a simple database transaction. Maybe it's a bot countermeasure.

[Slave IV]

Lol, I guess it would make sense if their crappy site was purposely designed that way to deter scalpers...and all other people trying to shop there. 

[Sandman]

Both HLJ and Amiami changed their websites for the worst. I haven’t secured a preorder since they made this change last year. 

[Guyffon]

I think the only problem is with bandai DX stuff, i have had no problems preordering the HMR VF-4S recently from amiami, and even the AFC-01H from amiami 8 days after it came out for preorder. I think most of the problems stem from scalpers and their bots querying and putting the DX stuff to cart that freezes up their system.

[Nerd-linger]

My technical knowledge is very limited, but I wonder why places like HLJ don't use anti-bot reCaptcha (select all the squares with a car... etc) as part of the checkout process. Maybe todays bots are too advanced?

[Sanity is Optional]

When things go out of stock within minutes, or even less than a minute, and the website's scripts are unresponsive in general... Well it would lead to a lot more complaining than is currently the case.

[Nerd-linger]

One has to wonder how much of that traffic are bots. If hlj could find a way to make bots useless or in the very least make it difficult enough for scalpers choose "easier prey" perhaps the remaining traffic would be manageable. 

[SpaceCowboy]

Here was my experience writing a preorder bot....

Shortly after this thread popped up, I thought, "hell, I'll give it a shot", and wrote one that would go through HLJ all the way through payment and actually buy something.  It handled order-stop, would auto search for terms, would wait for the "add to cart" button to go live.  The whole shebang.  Then I forgot about it for months until the Roy preorder popped up.  I pulled it out, everything was still working...

Then HLJ completely redid their site a few days ago.

:|

So I did a few fixes to make it work again, but everything was slightly different but kind of the same...but I got it working, but some of my corner cases I couldn't test thoroughly (due to lack of time on my part).  My bot did indeed find the VF-1S Chogokin.  Except HLJ had decided that even if you couldn't "add to cart" it was going to show the button anyways.  So, I might have been able to program around this quickly if this was my livelihood, but as is, I just wanted my one toy, and I was up watching the bot, seeing if it would work, and on another system was refreshing amiami.  So my time was better spent just trying to make sure my amiami order went through.

My take away from this is:

* for every site that you want a bot for, you need to really understand their purchase flow control and be able to handle it individually.  Expect that they will change their HTML responses.

* If the community paid someone, they would probably be up, preorder night, dealing with last minute changes or cases they didn't know about (amiami deciding to log you out randomly, or HLJ having a known page that doesn't load for 10 minutes).  Then you, as the customer, would need to take their latest changes as they made them (unless you want to trust random programmer with your CC# or account info)

* I probably spent about 8 hours total.  Let's say I'm an entry-level web developer making $27,000/year ($13/hour).  That is still over $100 of time I've spent - and I didn't get Roy with my bot.  I'm sure a pro could do it better/faster, but their time will also be more expensive.  Multiply whatever amount you come up with by the number of sites you want a bot for.

* My goal was to be able to sleep through preorder madness and wake up nice and refreshed with a purchased item.  It's clear that I'd need to be monitoring the bot and programming quickly to keep up with last minute changes.  Then if *I* were the person you were paying, most of the site would be pissed at me because the bot failed in one random direction or another.

Anyways, sharing my perspective.  It was a fun programming exercise, and it's not terribly complicated, but in the end, it is extremely fragile due to working on a black box that is subject to change and has conditions that are difficult to test for.

Good luck to everyone still searching.