Python AH Buyer / Seller / ffxiah.com scrubber

Applications/Tools written to make running DarkStar easier for the rest of us.
Desufire
Posts: 162
Joined: Sun Feb 22, 2015 2:58 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Desufire » Wed Mar 25, 2015 5:39 pm

I've decided I'd write a little "How to" guide on how to set this up and at least get it running. I spent an entire day troubleshooting and trial and error in order to get it working how I do. This is not the best way to run and I'm hoping someone who uses this program correctly will read this and correct any shortcuts I may have taken. Until then though, I do have this working and items are being stocked.

This guide is for Windows.

First thing's first. Go here https://www.python.org/downloads/windows/ and scroll down until you see Python 2.7.9 - 2014-12-10. Under this you will see 2 MSI installers. Choose the "x84 MSI installer" if you run Windows 32-bit. Choose the "x84-64 MSI installer" if you run 64-bit. Once the download is finished, you can close that page, we're done with it.

Install Python onto you C: drive. (You could install it anywhere, but I did it this way to make a later step a little easier. Up to you.) The install path will look like: C:\Python27.

Once you're done installing, you need to set the Path for Windows, which is what a lot of people don't realize who are new to this sort of thing.

Click Start Menu, then Right Click on My Computer or Computer. Choose Properties. You should have a Control Panel window come up showing you your computers information. On the left you'll see Advanced System Settings. Click that.

On the bottom of the new window, click Environment Variables. You should have a small window showing 2 scroll boxes. The top box is User variables for MyPCName and the bottom box is System Variables. We want the bottom box.

Scroll the bottom box (System variables) down until you see the word Path on the left side. Click on Path which should highlight the entire line. Now click Edit... at the bottom.

Now we need to type in the install path for Python27 and Python27 Scripts. If you installed it in your Program Files under your C drive, your path will look like::
C:\Program Files\Python27;C:\Program Files\Python27\Scripts
Easy way to see what your path is, is to open the folder where Python27 is, then click the address bar on the top.

You need to do both Python27 and the Scripts folder in Python27 folder. Be sure to use semicolon in between each path like I did above.

When you're done, click Okay on all windows. You can close any other window because the hard part is done.

Now we just use pip to install the 4 other programs the author lists on his git page https://github.com/AdamGagorik/pydarkstar.

To use pip, click your Start Menu and type cmd in your search bar. This will bring up the Windows Command Prompt. \

If you get any errors during the next steps, you will need to solve them before continuing. The common one will be missing C++. If you see this, all you have to do is google for microsoft visual for python and get the redistribution package for python.

In your Windows cmd prompt type: pip install sqlalchemy then hit enter. Wait for pip to install the program and look for any errors during the install. Even if there are errors, pip will state the install was successful. When install is done, we do the same for the following 3 programs needed.
pip install pymysql
pip install beautifulsoup4
pip install pyyaml


We're ready to gitpull pydarkstar now. You can go to the git address I linked above and download the zip or perform a pull like when you were setting your server up. You can install pydarkstar anywhere you want.

Once you have the folder placed and open, you'll see a file called "makebin". Double click this and it'll create a bin folder.

Now you can go into the apps folder and open the broker file using your file editor (I use Notepad++). There are a few options to change here.

Code: Select all

# input
        self.data     = []    # list of itemdata
        self.find     = True # search for item data --This will be set to False. Change to True.

Code: Select all

# sql
        self.hostname = '127.0.0.1'
        self.database = 'dspdb'
        self.username = 'root'
        self.password = '' --Insert your password for MySQL/Server
        self.fail     = False # fail on SQL errors

Code: Select all

self.name     = 'Caladbolg' # seller name --Caladbolg is my server name. Your script will show something else.
        self.restock  = 3600     # restock tick
        self.refill   = True    # restock at start --Your script will show False. Change it to True.
Now for the long and tedious part and the point where I'd like someone more familiar with this to assist. In the main pydarkstar folder, there's a data folder with the files; alter, broker, items.csv, scrub. The items.csv has a list of most every item that can be sold on the AH with prices adjusted to a specific server. What I did was to open this items.csv file in Notepad++ and went through every item I did NOT want stocked in my AH. Then I adjusted prices on certain items. This takes a LONG time.

Once you're done, save your file in the bin folder! Do not overwrite the items.csv in the data folder!

Once saved, just double click and run the broker file in the bin folder. Let it run until the command prompt disappears. Congrats. Your AH is stocked.

As I stated at the start, I have not been able to figure out how to run this the proper way. I know in my gut that my way is a work around and uses this program in a half assed way, but it's what I've been able to figure out. I hope this helps others get it working and I hope that someone will see this and help me understand how to use it the way it's meant to be used.

Thank you.
Last edited by Desufire on Wed Mar 25, 2015 7:51 pm, edited 1 time in total.

jankenpoy
Posts: 11
Joined: Fri Feb 27, 2015 11:26 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by jankenpoy » Wed Mar 25, 2015 6:57 pm

First of all, thank you very much Desufire! I did try to follow your guide, unfortunately for me at the end I still got an error: "RuntimeError: missing item data CSV!" after I run the broker file in the bin folder. I did saved a copy of items.csv in the bin folder, but really thanks for the guide, I got to this point and I'm sure I'm almost there!

Desufire
Posts: 162
Joined: Sun Feb 22, 2015 2:58 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Desufire » Wed Mar 25, 2015 7:52 pm

Yeah, I missed a step. I just finished editing the guide to include it. You have to change self.find = False to self.find = True.

This'll make the broker script find the items.csv

jankenpoy
Posts: 11
Joined: Fri Feb 27, 2015 11:26 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by jankenpoy » Wed Mar 25, 2015 8:35 pm

okay! now its working. thank you so much Desufire! hopefully in the future, someone will make a tool for the number of stock according to ffxiah.com.

Desufire
Posts: 162
Joined: Sun Feb 22, 2015 2:58 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Desufire » Thu Mar 26, 2015 1:01 am

In the items.csv you can adjust the amount an item is stocked for single and stacks. You can also adjust the prices for each as well.

For example, to adjust the amount of Fire Crystal stacks that will be placed on the AH:

Open items.csv, ctrl+F, type fire crystal to find it's location faster. Once there, scroll to the right and you'll see two lines that have the number 5. The first line is for single stock while the 2nd is for stack stock. So if we wanted 50 single Fire Crystals and 20 stacks of Fire Crystals to be stocked, we'd change those 5's into 50 and 20. Then save the file and run broker. If you have the self.refill = True, it will immediately change the stock to 50 and 20. Otherwise it'll do it an hour from when it last ran.

Desufire
Posts: 162
Joined: Sun Feb 22, 2015 2:58 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Desufire » Mon Mar 30, 2015 6:35 am

Code: Select all

  itemid,                     name, sell01,  buy01,          price01, stock01, sell12,  buy12,          price12, stock12
   16690,                 Cesti +1,      1,      1,             1000,      10,      0,      1,                1,       5
   17476,         Cat Baghnakhs +1,      1,      1,             1000,      10,      0,      1,                1,       5
   16440,       Bronze Knuckles +1,      1,      1,             1000,      10,      0,      1,                1,       5
   16689,        Brass Knuckles +1,      1,      1,             5000,      10,      0,      1,                1,       5
   16691,                Ryl Cesti,      1,      1,            15000,      10,      0,      1,                1,       5
   16442,                 Fsd Bagh,      1,      1,            15000,      10,      0,      1,                1,       5
   16433,                Lgn Knuck,      1,      1,            15000,      10,      0,      1,                1,       5
   17519,			 Platoon Cesti,		 1,		 1,			   25000,	   10,		0,		1,				  1,	   5
   16492,         Bronze Dagger +1,      1,      1,             1000,      10,      0,      1,                1,       5
   16491,          Bronze Knife +1,      1,      1,             1000,      10,      0,      1,                1,       5
   16740,          Brass Dagger +1,      1,      1,             5000,      10,      0,      1,                1,       5
   16736,				 Dagger +1,		 1, 	 1,				8500,	   10,		0,		1,				  1,	   5
   16614,                 Knife +1,      1,      1,             9000,      10,      0,      1,                1,       5
   17993,			Platoon Dagger,		 1,		 1, 		   25000,	   10,		0,		1,				  1,	   5
   16745,			Decurion's Dag,		 1,		 1,			   25000,	   10,		0,		1,				  1,	   5
   16746,			Mercenary Knif,		 1,		 1,			   25000,	   10,		0,		1,			 	  1,	   5
   16473,                    Kukri,      1,      1,            25000,      10,      0,      1,                1,       5
   16610,             Wax Sword +1,      1,      1,             8000,      10,      0,      1,                1,       5
   16544,                 Ryl Arch,      1,      1,            10000,      10,      0,      1,                1,       5
   17692,			 Platoon Sword,		 1,		 1,			   20000,	   10,		0,		1,				  1,	   5
   16632,                 Bilbo +1,      1,      1,            20000,      10,      0,      1,                1,       5
   16637,			  Deathbringer,		 1,		 1,			    5500,	   10,		0,		1,			      1,	   5
   16638,              Claymore +1,      1,      1,             7500,      10,      0,      1,                1,       5
   16930,			Mrc. Greatswor,		 1,		 1,			   20000,	    10,		0,		1,				  1,	   5
   16959,			Platoon Greats,		 1,		 1,			   20000,		10,		0,		1,				  1,	   5
   16646,            Bronze Axe +1,      1,      1,             1000,       10,     0,      1,                1,       5
   16661,             Brass Axe +1,      1,      1,             5000,       10,     0,      1,                1,       5
   16648,        Legionnaire's Axe,      1,      1,            15000,       10,     0,      1,                1,       5
   16663,             Battleaxe +1,      1,      1,             5000,       10,     0,      1,                1,       5
   16716,		 Butterfly Gaxe +1,		 1,		 1,			   10000,		10,		0,		1,				  1,	   5
   16717,              Greataxe +1,      1,      1,            30000,       10,     0,      1,                1,       5
   18209,			Platoon Cutter,		 1,		 1,			   30000,		10,		0,		1,				  1,	   5
   16778,			Bronz Zaghn +1,		 1,		 1,				1000,		10,		0,		1,				  1,	   5
   16780,			   Lgn. Scythe,		 1,		 1,			   10000,		10,		0,		1,				  1,	   5
   18045,		   Platoon Zaghnal,		 1,		 1,			   25000,		10,		0,		1,				  1,	   5
   16859,          Bronze Spear +1,      1,      1,             1000,       10,     0,      1,                1,       5
   16852,                Ryl Spear,      1,      1,            10000,       10,     0,      1,                1,       5
   16864,           Brass Spear +1,      1,      1,            10000,       10,     0,      1,                1,       5
   16914,                 Kunai +1,      1,      1,             1000,       10,     0,      1,                1,       5
   16918,             Wakizashi +1,      1,      1,             8000,       10,     0,      1,                1,       5
   16920,        Shinobi-Gatana +1,      1,      1,            10500,       10,     0,      1,                1,       5
   16981,				  Tachi +1,		 1,		 1,			    5000,		10,		0,		1,				  1,	   5
   16978,			 Uchigatana +1,		 1,		 1,			   10000,		10,		0,		1,				  1,	   5
   17820,				 Gunromaru,		 1,		 1,			   30000,		10,		0,		1,				  1,	   5
   17087,            Maple Wand +1,      1,      1,             2500,       10,     0,      1,                1,       5
   17111,            Bronze Rod +1,      1,      1,             1000,       10,     0,      1,                1,       5
   17028,         Freesword's Club,      1,      1,             5000,       10,     0,      1,                1,       5
   17139,               Solid Club,      1,      1,             5000,       10,     0,      1,                1,       5
   17462,             Platoon Mace,      1,      1,            10000,       10,     0,      1,                1,       5
   17123,             Ash Staff +1,      1,      1,             1000,       10,     0,      1,                1,       5
   17122,              Ash Pole +1,      1,      1,             1000,       10,     0,      1,                1,       5
   17130,        Freesword's Staff,      1,      1,            10000,       10,     0,      1,                1,       5
   17128,				Lgn. Staff,		 1,		 1,			   10000,		10,		0,		1,				  1,	   5
   17103,				Merc. Pole,		 1,		 1,			   30000,		10,		0,		1,				  1,	   5
   17571,				Platoon Po,		 1,		 1,			   30000,		10,		0,		1,				  1,	   5
   17175,              Shortbow +1,      1,      1,             2000,       10,     0,      1,                1,       5
   17228,			Light Crossbow,		 1,		 1,			    2000,		10,		0,		1,				  1,	   5
   17290,			Coarse Boomera,		 1,		 1,			    2000,		10,		0,		1,				  1,	   5
   19225,			  Musketoon +1,		 1,		 1,			    3000,		10,		0,		1,				  1,	   5
   17223,			 Lgn. Crossbow,		 1,		 1,			   25000,		10,		0,		1,				  1,	   5
   17159,          Freesword's Bow,      1,      1,            25000,       10,     0,      1,                1,       5
   17167,			Ryl.Arc. Longb,		 1,		 1,			   25000,		10,		0,		1,				  1,	   5
   17258,		   Bandit's Gun +1,		 1,		 1,			   25000,		10,		0,		1,				  1, 	   5
   18171,			  Platoon Disc,		 1,		 1,			   25000,		10,		0,		1,				  1,	   5
   18170,			  Platoon Edge,		 1,		 1,			   25000,		10,		0,		1,				  1,	   5
   17202,			   Platoon Bow,		 1,		 1,			   25000,		10,		0,		1,				  1,	   5
   17271,			   Platoon Gun,		 1,		 1,			   25000,		10,		0,		1,				  1,	   5
   17373,				Maple Harp,		 1,		 1,			    5000,		10,		0,		1,				  1,	   5
   17374,                  Harp +1,      1,      1,            10000,       10,     0,      1,                1,       5
   17367,			Ryl. Spr. Horn,		 1,		 1,			   35000,		10,		0,		1,				  1,	   5
   5337,		 Sleep Blt. Quiver,		 1,		 1,				5000,		50,		25,		1,			  50000,	   5
   5339,		 Blood Blt. Quiver,		 1,		 1,				5000,		50,		25,		1,			  50000,	   5
   5333,		 	  Sleep Quiver,		 1,		 1,				5000,		50,		25,		1,			  50000,	   5
   4219,		 	  Stone Quiver,		 1,		 1,				5000,		50,		25,		1,			  50000,	   5
   4220,		 	   Bone Quiver,		 1,		 1,				5000,		50,		25,		1,			  50000,	   5
   4221,		 	 Beetle Quiver,		 1,		 1,				5000,		50,		25,		1,			  50000,	   5
   4222,		 	   Horn Quiver,		 1,		 1,				5000,		50,		25,		1,			  50000,	   5
   4227,		   Brz.Blt. Quiver,		 1,		 1,				5000,		50,		25,		1,			  50000,	   5
   5363,		 	  Bullet Pouch,		 1,		 1,				5000,		50,		25,		1,			  50000,	   5
   534,		   Gausebit Wildegrass,		 1,		 1,				  10,		75,		50,		1,				100,	   5
   17016,		 	Pet Food Alpha,		 1,		 1,				 500,		50,		25,		1,			   5000,	   5
   17017,		 	 Pet Food Beta,		 1,		 1,				 500,		50,		25,		1,			   5000,	   5
   17877,		 	Fish Oil Broth,		 1,		 1,				5500,		50,		25,		1,			  55000,	   5
   12340,			 Marine Shield,		 1,		 1,			    5000,		10,		0,		1,				  1, 	   5
   12333,		   Lauan Shield +1,		 1,		 1,			    5000,		10,		0,		1,				  1, 	   5
   12337,		 Decurion's Shield,		 1,		 1,			   15000,		10,		0,		1,				  1,	   5
   5308,		  Toolbag Uchitake,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5309,		   Toolbag Tsurara,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5310,	  Toolbag kawahori-ogi,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5311,	     Toolbag makibishi,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5312,	     Toolbag hiraishin,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5313,	    Toolbag mizu-deppo,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5314,		    Toolbag shihei,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5315,		   Toolbag jusatsu,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5316,		  Toolbag kaginawa,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5317,		Toolbag sairui-ran,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5318,		    Toolbag kodoku,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5319,	  Toolbag shinobi-tabi,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5417,   Toolbag sanjaku-tenugui,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5734,		     Toolbag soshi,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5863,		   Toolbag kabenro,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5864,		     Toolbag jinko,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5865,		     Toolbag ryuno,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5866,		   Toolbag mokujin,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5867,   Toolbag inoshishinofuda,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5868,   	   Toolbag shikanofuda,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   5869,	     Toolbag chonofuda,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   6265,   	   		 Toolbag ranka,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5
   6266,	   		  Toolbag furu,		 1,		 1,			    5000,		50,		25,		1,			  55000,	   5

Above is my items.csv file.

I currently use the AH_Tool found here: viewtopic.php?f=16&t=15 to stock my AH with Crafting materials. The prices are fairly set (not cheap and not breaking the bank) and it saves me hours of having to insert them into an items.csv.

I use this pydarkstar AH tool to stock the above items. You can change the prices, add/remove items you want, and adjust the amount sold. You might want to straighten it up since it most likely won't copy and paste in an orderly fashion.

Lucious_Llama
Posts: 40
Joined: Mon Jul 28, 2014 5:22 pm

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Lucious_Llama » Thu Apr 02, 2015 9:42 am

Hi Guys -- I haven't checked this thread in some time, but I am glad to see people are using this. To more easily get my attention, open issues on GitHub.
I used to think I knew a good amount of coding until i started looking into this project and hosting a server.
Its not you, its Windows. Thanks for taking the time to write the guide on Windows setup. I am sorry it was so difficult to set up, I thought I designed it in a way to make things easy ><

I wanted to point a few things out that may be useful.
  1. There is a README, which you can view at https://github.com/AdamGagorik/pydarkstar, that contains a lot of setup and usage information
  2. I added a little section on setting things up on Windows to the README
  3. For an easier time on Windows, AKA avoiding all the PATH and ENVIRONMENT variables crazyness, I used the Anaconda Python distribution
  4. The official Python is just not friendly for people not used to command line environments
Now you can go into the apps folder and open the broker file using your file editor (I use Notepad++)
Rather than edit the python script itself, you should copy the yaml files from the data folder to the bin folder. I've altered the makebin script to do this copy step for you during initial setup.
The yaml files are like settings files. Open those up with Notepad++ and put the correct settings there.
You have to change self.find = False to self.find = True.
I am kind of regretting putting this setting in there now. You should actually pass the name of the CSV file to broker.bat as the first argument when running. Its best if you copy the items.csv to the same directory as your broker.bat script, and run .\broker.bat items.csv. I've altered the makebin.py script to do the copy step for you, during setup.

This may be difficult if you are double clicking on the bat script, instead of running it through the terminal. So keep using find=True if that is the case. Or, set the path to items.csv in the broker.yaml configuration file.
okay! now its working. thank you so much Desufire! hopefully in the future, someone will make a tool for the number of stock according to ffxiah.com.
While pydarkstar comes with an items.csv for you already (in the data folder), it was actually created from ffxiah.com using the scrub script. You can rerun this yourself to have up to date data. It just takes forever.
If you have the self.refill = True, it will immediately change the stock to 50 and 20
This is only one of the ways the broker script is meant to work. If you have the refill setting set to False, broker will run continuously as a server. It will buy items that players put up for sale, and restock items after players buy them. This refill mode is meant for a 1-time setup to stock the AH right away. Otherwise broker will wait the full restock time setting (3600 seconds = 1 hour) before initially restocking items.

Desufire
Posts: 162
Joined: Sun Feb 22, 2015 2:58 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Desufire » Fri Apr 03, 2015 12:40 am

Lucious_Llama, the biggest problem I had was not understand anything in the ReadMe file. After awhile, some tinkering, and a lot of google searches, I was able to grasp the concept of what your ReadMe and the steps listed on the github was saying.

Setting up the entire thing wasn't explained though, so the guide I made was what I was able to get working. Like I said in the post, I knew it was a half assed way of using the program lol.

Thank you for the post and tips though. I learn best from either "let me hold your hand thru this process" guides or just trial and erroring it myself lol. I'll go through what you said and see if I can't get this running properly now.

Delaide
Posts: 478
Joined: Sat Jun 14, 2014 8:58 am

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by Delaide » Fri Apr 24, 2015 4:22 am

Quick question about this. Does this basically collect a list of all items, or is it targeted?
I was wondering because I want a master list of all in game items to go through the DSP items sqls and see what is missing.

Only other way I can think of is to put together my own scraper, since at this point I do not know enough to modify the code, so would need to learn the whole process -.-

User avatar
TeoTwawki
Developer
Posts: 527
Joined: Mon Jul 15, 2013 9:50 pm

Re: Python AH Buyer / Seller / ffxiah.com scrubber

Post by TeoTwawki » Fri Apr 24, 2015 10:34 am

Delaide wrote:Quick question about this. Does this basically collect a list of all items, or is it targeted?
I was wondering because I want a master list of all in game items to go through the DSP items sqls and see what is missing.

Only other way I can think of is to put together my own scraper, since at this point I do not know enough to modify the code, so would need to learn the whole process -.-
It scrapes anything with a sales history on ffxiah.com, which will not be every item that exists. You are better off getting that info from the client itself, which there are multiple ways of handling. the simplest being browse the dats with polutils. There is ample documentation of the dat formats, if anyone wnats to try and build an auto scraper to make the relevant queries for the projects database files.
Hi, I run The Demiurge server.


Image
Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. - Martin Golding
PLS USE [ code ] CODE TAGS [ /code ] WHEN POSTING CODE
DO NOT PRIVATE MESSAGE ME ABOUT BUGS

Post Reply