Wasting Time
Ale, music and enjoying life.
Implementing paging with the Twitter Search API
Useful article here runs through the paging mechanism available through the Twitter search API
https://dev.twitter.com/rest/public/timelines
Super Furry Animals return
All this new Super Furry Animals activity has reminded me of the time I saw them at the O2 festival in Leeds mid 2000s sometime I guess. Great atmosphere in the tent and at the end it turned from a rock gig to a rave – Here’s some crappy footage I filmed
SSIS – Manually assigning Null values
Spent a few minutes trying to assign null to a derived variable as part of a script task transform. In my mind Row.CreatedDate = null would do it right?
After, making the changes and running the data through again I could see that this had no affect and all dates I thought I’d assigned null to had the DateTime.Min values. Not what I wanted.
After doing a bit of digging online I chanced upon the following post
As you can see all columns have their own Row.ColumnName_IsNull – Setting the relevant columns to True worked fine.
XML Refresher
Haven’t really touched XML for a fair few months now so by way of a refresher I worked through the excellent Pluralsight “Many Approaches To XML Processing” course. Notes below:-
Use XmlReader when reading LARGE documents from disk – does not read complete document. very fast – use when performance is an issue. You’d typically loop through each element reading (“ReadElementString”) into a stringbuilder.
XPath is more flexible – able to navigate and select nodes (“Similar to finding folders on a hard drive”) through XML.
Use XDocument (.Load) to read into memory complete XML document. You can use
XPath (XPathSelectElement(“/Employes/Employee”)
XPath (XPathSelectElements(“//Employes”) then foreach through each element or Linq to select nodes. Xml to Linq is easier to read than XPath
DataSet
Reads complete into memory – creates datatables
Read in XML document – loop through rows
Create a new DataSet then ReadXml on that object – Foreach through table’s rows. Can use search logic i.e ds.Tables[0].Select(“LastName LIKE ‘s%’);
ReadXmlSchema will validate xml against XSD.
SSIS is replacing null values with Zeros for integers
Hit an issue recently where a csv file was being transformed through an SSIS package but persisting to the database as zero value integers for all null value columns.
Spent time examining table constraints through trying to establish why this was occurring. It’s not the SSIS package because the file preview isn’t showing zeros right? Nope, guess again it is the SSIS package. Take a look at the Flat File flow component and tick the box as follows, job done.
