scripts for extracting data

[alterecco]

I know that most of this data is already collected here, but i needed an easy way to recreate it, so here some data from the vanilla xml, and a couple of bash scripts to extract them

Modifiers

Code: Select all

#!/bin/bash
OUTFILE=./TransModifiers
SOURCE=./TranscendenceSource/*

grep "modifiers=.*\"[[:alnum:],;[:space:]-]+\"" $SOURCE -ohE > $OUTFILE
sed -e 's/modifiers=//' -e 's/^[ \t]*//;s/[ \t]*$//' -e 's/"//g' $OUTFILE | tr ' ' '\n' | sed 's/;//' | sort | uniq | sed -e :a -e '$!N; s/\n/ /; ta' | sed 's/  / /g;s/^ //' > $OUTFILE

Result

Code: Select all

100MW Alien AntiMatter Armor ArmorEnhance Art Auton Barricade BlackMarketID Bushido CannotOrder ConfirmRefuel Consumable ContrabandBox Domina Dwarg EI EnergyWeapon FieldCrystal Food Fuel HaloGem Heliotropes ID Illegal Info LamplighterDataROM LamplighterPrototype LongzhuSphere Luminous Lux MajorItem Makayev Meds Military MilitaryID MinersHold MinorItem Missile NAMI NotForSale Nuclear Ore Psionic QuantumCPU Rasiermesser Res RingerValuable Sapiens SealedContainer Slaves SmugglersHold Solon Soul Specialty Taikon Urak ZeroPoint centauriWarlords qianlongArchcannon sungSlavers unknown

Attributes

Code: Select all

#!/bin/bash
OUTFILE=./TransAttributes
SOURCE=./TranscendenceSource/*

grep "attributes=.*\"[[:alnum:][:space:],-]+\"" $SOURCE -ohE > $OUTFILE
sed -e 's/attributes=//' -e 's/^[ \t]*//;s/[ \t]*$//' -e 's/"//g' $OUTFILE | tr ',' '\n' | sort | uniq | sed -e :a -e '$!N; s/\n/ /; ta' | sed 's/  / /g;s/^ //' > $OUTFILE

Result

Code: Select all

enemy envAir envEarth envFire envWater huari majorStation populated primaryStronghold sungSlaveCamp sungSlavers uncharted CSCEuropa KorolovCharon NewBeyond anarchists arcoVaughn ares aresMajor asteroid asteroidbelt asteroids auton benninXenophobes blackMarket blackmarket centauriWarlords charonFortress charonFrigate charonPirates comet commonFleet commonMilitary commonwealth corporate deathDrugsCartel debris desert drugcartel dwarg dwargRaiders enemy envAir envAvoidsAir envAvoidsEarth envAvoidsFire envAvoidsWater envEarth envFire envWater ferian ferianMiner fleet fleetDelivery freighter friendly frost gaianTerraformers gasgiant genericClass heliotropes himalSettlers huari huariTemple independent iocrym kobol kobolWarlords luminous luminousDrone majellen marauders methanegiant mining moon nebula nebulae newBeyond occupation outerRealm outlaw outlaws penitent pirates planet planetary populated primary raisuStation ranx religious ringedgiant rogue salvagers sapiens settlers stKatharine star sungSlaveQuarters trojan uncharted ungoverned urak urakWarlords ventari void volcanic

Ships

Code: Select all

#!/bin/bash
## SHIP UNID'S
TMPFILE=./TransShipTmp
OUTFILE=./TransShipUNID
SOURCE=./TranscendenceSource/*

grep "<ShipClass.*UNID" $SOURCE -hC1 > $OUTFILE
sed -e '/UNID="&.*;"/!d' $OUTFILE | grep '&[[:alnum:]-]+;' -Eo | sed 's/^&//;s/;$//' > $TMPFILE 
grep -hf $TMPFILE $SOURCE | grep 'ENTITY' | sed -e 's/<!ENTITY//' -e 's/^[ \t]*//;s/[ \t]*$//' -e 's/>$/)/' -e 's/\t*//g' -e 's/\([0-9a-zA-Z-]\+\)/(list \&\1; "\1" /' > $OUTFILE
rm $TMPFILE

Result
Too large to post here, so get it at the pastebin

Items

Code: Select all

#!/bin/bash
## ITEM UNID'S
TMPFILE=./TransItemTmp
OUTFILE=./TransItemUNID
SOURCE=./TranscendenceSource/*

grep "<ItemType.*UNID" $SOURCE -hC1 > $OUTFILE
sed -e '/UNID="&.*;"/!d' $OUTFILE | grep '&[[:alnum:]-]+;' -Eo | sed 's/^&//;s/;$//' > $TMPFILE 
grep -hf $TMPFILE $SOURCE | grep 'ENTITY' | sed -e 's/<!ENTITY//' -e 's/^[ \t]*//;s/[ \t]*$//' -e 's/>$/)/' -e 's/\t*//g' -e 's/\([0-9a-zA-Z-]\+\)/(list \&\1; "\1" /' > $OUTFILE
rm $TMPFILE

Result
Too large to post here, so get it at the pastebin



There you go. The scripts are regular shell scripts, using grep, sed and tr. They should run on any linux and cygwin.

.]

[alterecco]

Well, the actual reason for building the scripts above, was to build something I though would take more work. So, I hope that Bimbel will get a hold of this, and build a nice browseable page of this info.


Basically, it is a Tscript that extracts as much data as possible about ships, and items and spits it out as csv data, which has then again been converted into sql.

I will be posting both the Tscript and the sql as pasties, so dig in ppl.

Tscript

Note that the output csv needs a little bit of cleaning up before it can be imported.

sql

This is a long file, and won't make much sense unless you have a db server to import it into. It is dumped from mysql 5.1, and is as normalized as i could come up with.

For a quick overview of the db structure, here is another pastie

Ok, i'll stop with the pasties now. Have fun

.]