Being sick last week caused a few problems in getting some items on my task list completed on schedule.  The good news is there were some other site issues that came up during the week that I had been putting off for a long time, which are being resolved now.  You probably noticed some of the changes and not the rest.  The change of the category names for Uppers was likely seen by some.  Some of the parts moving from the wrong categories to the correct ones may have been caught by some as well.  Larger back end changes this weekend were most certainly missed.  This leads to the main topic for this update…

Continue reading →

I have had a lot of people ask what it takes for the site to do what it does.  Even though I touch on a few points in the FAQ, I figured a quick write up wouldn’t hurt.

Likely not surprising, the site runs on a dedicated VM with a database at my hosting company, with all the pages being dynamically created when you request them… much like many sites you visit.  Where we diverge from most of those other sites is the additional VM dedicated to handling site scraping/feeds and submitting those results to the previously mentioned database.  The whole thing was custom written from scratch based on a quick pseudo-code block I wrote in Emacs (think notepad, but for unix) the first week.  That same write up turned into a longer to-do list, which is still very long with all the future stuff I want to implement.

By the way, I typically use the word “scan” when I mention adding a new retailer on the front page news list.  I think the word makes the most sense as the system is scanning pages on the retailer site similar to how you scan the page in a magazine, looking for what is important.  Really, though, I grab the html (the stuff you see when you right click and View Source) like a web browser does and parse the text for the information.  This means I am not requesting images, which would waste bandwidth for both the retailer and myself.  Some people call it data mining and others call it site scraping.  Tohmato – tomahto.

Interesting notes about the site/scanner:

• Retailer scans and db updates take about 700 gigabytes per month in bandwidth currently.
• The web site uses only about 50 gigs a month, due to external thumbnails and minimal images.
• There are typically between 15 and 20 retailer scans running every second.
• I typically have to fix at least 1 retailer’s site scan every 2-3 days as they tweak their site layout/html.
• Pauses between scans range from 1 minute to 15 minutes (typically long delays for ones who don’t change stock often).
• Some retailers are providing custom feeds, which ensure we catch all their products even if they change their site.
• Most new scan additions are done near midnight, to stop the New Items page from being spammed during busy times.

Don’t know if anyone found this stuff interesting.  Either way, I typed it all out… so there you go.

Added Hamlund Tactical to the site.  They had a bolt carrier (not complete BCG) in stock until literally seconds before I did the final scan for inclusion after testing.  Bummer.

Of note is the merging of Lower Build Kits into the Lower Parts Kits area.  I was hoping to find others, besides Palmetto State Armory, that included all the components needed… but it appears PSA is somewhat unique currently.  So, it seemed silly to have basically a PSA only part category.

Also, I added Heroes Tactical (why is everyone so tactical?) Friday.  I know they had 25 BCGs in stock and their inventory is live (despite the fact you can add 100 to your cart).  It seems they sold those 25 out, likely due to ARPartsFinder, by Saturday night.  I was told they expect more in this coming week… so keep an eye out.

Preference for how many items to show per page should now store your choice for 3 years and not till the browser session ends.  Thanks Kory H. for calling behavior that out.

Finally… I have been working on even more stuff that should go live before the end of the week.  I will keep adding other sites in the meantime, but I did want to give a heads up!