Recent Insights

Instagram x Amazon Reviews: The Difference Between Being a Collaborator and Being a Target

Instagram x Amazon Reviews: The Difference Between Being a Collaborator and Being a Target

373 Followers I haven’t been keeping up with @zoraflorasays lately so I haven’t had much to write about, but with an increase in Amazon reviews I’ve picked it back up (at least for a while). Over the long weekend I realized that when you’re doing […]

Instagram x Amazon Reviews: Using Instagram to Get Sellers’ Attention

Instagram x Amazon Reviews: Using Instagram to Get Sellers’ Attention

339 Followers Amazon Ranking: #30,795 On June 29th, I wrote about how social media helped way less than I expected. I stand by that; most of my requests come through my email account. I wanted to expand on it though because “not helping” doesn’t explain […]

Data Conversion Between XML and JSON (w/ Scala)

Data Conversion Between XML and JSON (w/ Scala)

I don’t write technical posts often because I have had to dig through more than my fair share of irrelevant blog posts from 2007. However, every once in a while I run across a problem that has a  solution so convoluted and inconvenient that I feel the need to put it in one place; so here we are.

Disclaimers:
I have been working in Scala for the past year or so (but am not an expert or particularly fond of it). If you haven’t touched Scala before, it’s very similar to Java but the syntax is not exactly the same (and also it’s a functional language, so you end up creating a lot of vals). I modified a lot of Java snippets to get this working in Scala. It is not perfect Scala, if you have a better solution, feel free to share it.
Also, I’ve pulled these code snippets out of a much larger project from multiple files, you will run into issues if you try to just copy/paste them.

The Problem

If you’ve ever had to handle data, then you know the biggest headache is in the structure of it. JSON and XML are both ways to describe and structure how data is organized. XML came to be around 1996, JSON was created in the early 2000s. In my experience, JSON is generally seen as the more “modern” and usable approach – this is the data format used most often in Javascript – but some people are still diehard XML fans.

If you’re working with third party API’s that have been around for a while then there’s a pretty good chance that they’ll be returning XML when you’re using JSON. If you’re unfamiliar with both formats, let me try to explain why this is what’s been keeping me up for the past five nights.

XML

<root>
	<person>
		<name>Jones</name>
		<age>23</age>
		<occupation>Consultant</occupation>
	</person>
</root>

JSON

{
	person: {
		"name": "Jones",
		"age": 23
		"occupation": "Consultant"
	}
}

Right off the bat you can see that they’re organized a little bit differently. XML uses tags, JSON uses brackets and semicolons. The examples above are extremely simple and can pretty easily be parsed by libraries like LiftWeb, no big deal.

Unfortunately, in real life, the examples are rarely so simple, your XML is probably going to look a little bit more like this

<root>
	<People>
		<Person id=1>
			<PersonalDetails>
				<Name>Jones</Name>
				<Age>23</Age>
				<Locations>
					<Location reason="Born">Alaska</Location>
					<Location reason="Work">Texas</Location>
					<Location reason="Study">Moscow</Location>
					<Location reason="Study">Beijing</Location>
				</Locations>
			</PersonalDetails>
			<WorkDetails>
				<Job>
					<JobTitle>Consultant</JobTitle>
					<Company>Credera</Company>
					<HireDate>2016</HireDate>
					<EndDate></EndDate>
					<Skills>
						<Skill expertise="1">Scala</Skill>
						<Skill expertise="3">Javascript</Skill>
						<Skill>AngularJS</Skill>
					</Skills>
				</Job>
				<Job>
					<JobTitle>Personal Assistant</JobTitle>
					<Company>SMU</Company>
					<HireDate>2014</HireDate>
					<EndDate>2016</EndDate>
					<Skills>
						<Skill>Dewey Decimal System</Skill>
					</Skills>
				</Job>
			</WorkDetails>
		</Person>
	</People>
</root>
				
				
				

Suddenly there’s stuff inside the brackets, but not all of them (those are called attributes), some of the tags don’t have anything between them, and the XML is a lot less readable. The equivalent JSON might look like this:

{ "people": [
	{
		"id": 1,
		"personalDetails": {
			"name": "Jones",
			"age": 23,
			"locations": [
				{"reason": "Born", "location": "Alaska"},
				{"reason": "Work", "location": "Texas"},
				{"reason": "Study", "location": "Moscow"},
				{"reason": "Study", "location": "Beijing"}
			]
		},
		"workDetails": {
			"jobs": [
				{
					"jobTitle" : "Consultant",
					"company" : "Credera",
					"hireDate" : 2016,
					"endDate" : undefined,
					"skills": [
						{"expertise": 1, "skill": "Scala"},
						{"expertise": 3, "skill": "Javascript"},
						{"expertise": undefined, "skill": "AngularJS"},
					]
				},
				{ 
					"jobTitle" : "Consultant", 
					"company" : "Credera", 
					"hireDate" : 2016, 
					"endDate" : undefined,
					"skills": [ 
					    {"expertise": 1, "skill": "Scala"}, 
						{"expertise": 3, "skill": "Javascript"}, 
						{"expertise": undefined, "skill": "AngularJS"}, 
					] 
				}
			
			]
		}
	}
]}

You can see how the similarities between XML and the JSON models start to fall apart due to convention as the model gets more complicated. In XML you might have a variable data model (the types of data can change according to what data is available) but in JSON you will rarely run into a key that has a value that is occasionally a list, often a string, and sometimes just doesn’t exist, all on the same API call.

Parsing XML into JSON

For this first part we used scala.xml.NodeSeq to extrapolate the information we wanted and place it into objects accordingly.

  • You can pull out nodes by using the \ followed by the node name
  • You can pull out attributes by using \@ followed by the attribute name
 def toJSONObject(xmlObject: NodeSeq) : List[Person] = {
     val listOfPeople = new ListBuffer[Person]
     val people = xmlObject \ "People"
     
     people.map { person =>
       val personId = person \@ "id"
       val personalDetails = person \ "PersonalDetails"
	   val personName = personalDetails \ "Name"
	   val personAge = personDetails \ "Age"
	   
	   val workDetails = person \ "WorkDetails"
	   val jobs = workDetails \ "Jobs"
	   
	   val jobList = new ListBuffer[Job]
	   jobs.map { job =>
	   		val title = job \ "Title"
		    val company = job \ "Company"
	        jobList += Job(jobTitle = title, companyName = company)
	   }

       listOfPeople += Person(id  = personId.text, name = personName.text, age = personAge.text, jobs = jobList.toList )
       
     }
     listOfPeople.toList
  }

After we figured out how to do it, this became simple and even enjoyable. scala.xml.NodeSeq allows us to walk down the XML tree structure, grab the exact text and attributes that we want, then reformulate them in JSON objects that we’ve defined. If the node is blank or doesn’t exist, it returns an empty string instead of a parsing error. You just have to make sure that in your pre-defined JSON objects that every field is an Option[].

Voila, problem of parsing the weird ambiguous XML structure has been solved.

Parsing JSON as XML

This is where it gets weird. Unfortunately, it seems like it’s a lot harder to make elegant code that reliably parses your complex JSON objects back into XML.

The Scala Elem type that’s found in scala.xml._ allows you to create XML structures and mix in values in an incredibly simple way:

val person = Person(id = 1, name = "Jones", age = 23, occupation = "Consultant")

val xmlObject = 
	<root>
		<people>
			<person>
				<id>person.id</id>
				<personaldetails>
					<name>person.name</name>
					<age>person.age</age>
				</personaldetails>
				<workdetails>
					<title>person.occupation</title>
				</workdetails>
			</person>
		</people>
	</root>
				

If you’re dealing with elements that have lists or fields that may or may not exist, then Elem isn’t going to cut it. You want something that can parse your JSON into XML with attributes and a minimum amount of typing on your part. A lot of the libraries will cleanly parse JSON objects into XML even if they have complex organizations, but it was a struggle to find a library that would also dynamically parse attributes.

Staxon

This is where the Staxon library comes in (you can find GitHub documentation here). They have examples on their wiki for converting XML to JSON and JSON to XML so I won’t steal their thunder by copy and pasting their exact code here – but I will show you what we did.

Staxon solves the attribute issue by changing the way you name the keys in your JSON objects. @Symbols denote a key that is an attribute for the containing key (so in the example below, if you had a list of jobs the XML would look like <job order=2><title>Con... etc. etc.</job>

{
	"person": {
		"@id" : 1,
		"name" : "Jones",
		"job" : 
		[{
			"@order" : 2,
			"title" : "Consultant",
			"company" : "Credera" },
		 {
			 "@order" : 1, 
			 "title" : "UX Consultant", 
			 "company" : "New Economic School of Moscow" }
		 }
		
		]
	}
}

Unfortunately, Scala being the finicky beast that it is, you can’t use an @ symbol as the beginning of a key name in a JSON object. If you use single quotes (`) to escape the @ symbol your IDE will likely not give you any errors, but it will probably throw a runtime error. Our way around this was to add underscores ( _ ) in the model where we wanted the @ symbol to be, and then when we stringified the object we simply did a replace.all('_', '@') to get the desired format.

We also modified the Input and Output streams (originally Java inputStream and outputStream) from the original Staxon documentation into ByteArrayInputStream/ByteArrayOutputStream so we could pass in and parse out Strings instead of just printing to a file or the command line.

Disclaimer 2.0: To re-emphasize before I get 50 code reviews, this snippet is not code complete –
we declare implicit values of objects, translators, and jsonformatters with Spray in other files in our code.
The base of this function is usually intended to return the result of an API call, not to just transform an object (That’s where the Future()) comes in at the end.
The functionality of this snippet is spread out over at least 4-5 files and multiple functions
– I ordered it this way for simplicity in reading, not for efficiency.
import javax.xml.stream.XMLEventReader;
import javax.xml.stream.XMLEventWriter;
import javax.xml.stream.XMLOutputFactory;
import javax.xml.stream.XMLStreamException;

import de.odysseus.staxon.json.JsonXMLConfig;
import de.odysseus.staxon.json.JsonXMLConfigBuilder;
import de.odysseus.staxon.json.JsonXMLInputFactory;
import de.odysseus.staxon.xml.util.PrettyXMLEventWriter;

import java.io.ByteArrayInputStream
import java.io.ByteArrayOutputStream
//This may not be the complete list you need ^ so don't hate me if you still have to import some other libraries

def editPerson(person: Person): Future[Unit] = {
   
	//This toPerson() is a different transformational function where we add in default attribute values like the namespace, not included in this snippet 
    val formattedPerson = toPerson(person)
	
	/* EDIT: Because this has been mentioned in the comments
 	* We put this object into a JSON format - This JSON -> String -> XML can (and should) be put into a separate modularized function, 
	* I'm laying it out this way so you can see the linear process and not have to jump between functions
	* We use implicit values in order to get the "toJson" to work (case class Person() ) etc. 
	* 
	*/
    val jsonPerson = formattedPerson.toJson
    
    //We stringify the JSON format and replace all the _ with @ signs to indicate an attribute
    val stringPerson = jsonPerson.toString().replaceAll("_", "@")
		
	//The input is established as a ByteArrayInputStream (this is so it works with the Staxon methods)
    val input =  new java.io.ByteArrayInputStream(stringPerson.getBytes)
	
	//We send it to a translator that parses it to XML
    val requestBody = toXml(input)
		
	//We have to add the content type to the httpEntity before sending it off
    val httpEntity = HttpEntity.apply(MediaTypes.`application/xml`, requestBody)
		
	//Attach it to the request and get the result	
    val request = Put("/api/call?action=edit", httpEntity).withHeaders()
    val result = pipeline(request)
    
   Future()
  }


//This is a modified version of what is on the Staxon GitHub to allow for Stringification
def toXml(json: ByteArrayInputStream): String = {
    
    val config = new JsonXMLConfigBuilder().multiplePI(false).build();
    val output = new ByteArrayOutputStream();
		try {
			/*
			 * Create reader (JSON).
			 */
			val reader = new JsonXMLInputFactory(config).createXMLEventReader(json);
			
			/*
			 * Create writer (XML).
			 */
			val writer = XMLOutputFactory.newInstance().createXMLEventWriter(output);
			val prettyWriter = new PrettyXMLEventWriter(writer); // format output
			
			/*
			 * Copy events from reader to writer.
			 */
			prettyWriter.add(reader);
			
			/*
			 * Close reader/writer.
			 */
			reader.close();
			writer.close();
			val finalOutput = output.toString()
			finalOutput
		} finally {
			/*
			 * As per StAX specification, XMLEventReader/Writer.close() doesn't close
			 * the underlying stream.
			 */
			
			json.close();
			output.close();	
		}
  }
Copying JSON to XML via StAX Event API (Modified)

There you go. It’s not the prettiest way to parse something with all of the transformations, but trust me when I say it is super effective

Helpful References

Also, a large amount of credit goes to the technical lead on my project who did a lot of research and was the one who eventually found the Staxon library. When I say “we” in this article, the research that went into finding and implementing this solution was truly a team effort.

Instagram: The Digital Halo Effect

Instagram: The Digital Halo Effect

7,102 Followers In the flesh and blood world, the halo effect is a psychological phenomena that can either be annoying or helpful, depending on who you are: Halo Effect: The halo effect is a cognitive bias in which an observer’s overall impression of a person, company, […]

Photography: Mallory at the Arboretum

Photography: Mallory at the Arboretum

The Arboretum is by far one of the most picturesque venues you can find in Dallas. From the abundance of pergolas to secluded benches tucked away in private wooded areas to the arches formed from trees and bushes, it’s about as outdoorsy as you can […]

Instagram: The Perks of Leveling Up (5k followers)

Instagram: The Perks of Leveling Up (5k followers)

Followers: 6,147

Close to the beginning of my project, I talked about what a business account on Instagram will get you (namely, statistics). As your account grows, it seems like Instagram will “level you up” in terms of what you can/can’t do.

Here’s the list so far:

  • At 100 followers you can start seeing subscriber statistics
  • Around 5,000 followers I noticed that now in my stories I can see a lot more statistics. I checked this against my smaller business account (less than 500 followers) and the smaller account doesn’t have this capability.

Instagram Story Statistics

Everybody can see which users have seen their stories, but the new statistics are a step up:


With this new update you can now see:

  • Total times the story has been views (469)
  • The number of users who have looked at it (350)
    • If you subtract this number from the total story views you get “People who have looked at this story a second time”
  • The number of people who have directly responded to it (1)
  • The number of people who have clicked through the story before it finished running (71)
    • Don’t take this number personally, I know I usually flip through everyones stories, just because I read fast and don’t like waiting
  • The users/locations/hashtags tagged in the post and number of people who have clicked-through to the respective profiles (16)
    • This is a good measure of engagement

It seems like around 30,000 followers (just a guess based on observation) you can start to add links to Instagram stories; I’m looking forward to seeing what other little perks are out there as my follower count grows.

 

Note 8/30/17:  Instagram updates their UI fairly regularly. At the time of publishing Instagram seems to have removed this feature as I can no longer see it on either of my accounts. Maybe they’ll bring it back, maybe they’ll keep us in the dark

SaveSave

Instagram: Sometimes Negotiations Go South, and that’s Okay

Instagram: Sometimes Negotiations Go South, and that’s Okay

Followers: 6,038 Last week I posted about making my first Instagram collaboration for pay, but they don’t always go so smoothly. Soon after that company contacted me I was contacted by another person looking for a post-for-pay type deal. The Initial Approach It was going […]

Amazon Reviews Milestone: Reviewer Ranking Under 50k

Amazon Reviews Milestone: Reviewer Ranking Under 50k

Reviewer Ranking: #36,658 I started doing Amazon Reviews in November of 2016. My ranking at the time was past the 7 million mark (Amazon has a lot of customers). I started tracking my reviewer ranking in March, when I reached a rank of #190,607 and […]

Instagram: I Guess I’m an Influencer Now

Instagram: I Guess I’m an Influencer Now

5,774 Followers

First of all: The fact that over five thousand people care about pictures of me and my computer completely baffles me

Second of all: Today I was approached by a company who wants to pay me to make posts about their company

Third of all: I’ve since had two more companies contact me (unbeknownst to them, unless they all hire the same people) asking me about marketing and prices. I guess 5k-6k is the magic number for people starting to pay attention to you.

My first thoughts were something like, “Did they message the wrong person?,” “Did someone’s cat take over their phone?” “Is this a butt-dial?” After I calmed down enough to figure out how to respond, I realized that there was a huge void of information on what someone’s supposed to do in this situation (despite the fact that this is a rampant practice).

How much should I ask for?

How much should they ask for?

Do we need a legal agreement?

How am I supposed to act?

Is this even real?

So I did what I always do in these situations: research.

The Research

I found an interactive map at influencer.co that gives you the average prices for an influencer depending on gender, number of followers, engagement rates, location, and a few other categories. It seemed like for a following of my size I should definitely ask for less than $100 dollars.

I also read a lot of articles about people starting Instagram campaigns. Most of them are aimed at people who are trying to hire influencers, not people who are selling their photography/influence/opinion to others. I’ll tell you right now that most of those articles just go back and reference influencer.co’s information because apparently they’re the only people on the internet who have tried to actually quantify this stuff. Or at least they’re the only people who have gotten any credit for it. Some of those articles reference people getting paid $25-$75 per thousand followers.

I like the idea of having a ratio of money per thousand followers agreement because saying this up front prevents what I’ve often seen in dog-sitting: the awkward “I thought it was less last time” effect when you decide you and your time are worth more than $10 an hour.

Their message was in my inbox, burning a hole in my brain, waiting for me to respond with an appropriate number that neither insulted them nor devalued me. You never want to be the first person to put down a number, but they’d already admitted to being new to this and I didn’t want to get into an awkward “No, you first” conversation.

The Execution

I settled on $10/thousand rounding down. It would put my first post at $50 and would allow me to say “Hey, I have 2,000 more followers than last time, that makes me worth more to you” in the future.

They originally wanted five posts over a two-week period and I recommended extending that so they wouldn’t just be advertising to the same 5k people for two weeks when they could advertise to an additional 1k-4k if they waited another month. They said they’d have to talk to their team, I said “Sure!” and laughed nervously to myself, hoping no one would realize that I was just toddler wearing grown up clothes pretending to know what I’m doing with a bunch of statistics.*

*To be fair to myself, I actually did do a lot of research and it’s a legitimate marketing tactic – it just feels silly (and awesome) to be getting paid to take pictures of myself in my pajamas on my computer

They countered with an offer of $7/thousand followers for two posts over a two-week period with the idea that if they wanted to continue doing influencer marketing after that we would up the price. I agreed because, hey, that is more than the $0/thousand followers I usually make when I post things (and they were giving me access to a tutorial for free). I also agreed to it because I personally don’t know how effective it’ll be for them and I’d love to develop a working relationship with them in the future.

I’m happy my first time was with a company that seemed equally inexperienced and I’m excited to see how it works out (hopefully for the best for both of us).

Now, I have some other emails to respond to.

 

SaveSave

Amazon Reviews: Let’s Talk Algorithms

Amazon Reviews: Let’s Talk Algorithms

Amazon Ranking #51,485 After doing this for a little over six months, I’ve pieced together some bits about the Amazon ranking algorithm. There’s no way they update the ranking every single day There have been periods where I’ve gone up to 32 days without my […]