Big Data wrangling: "How to cram a skyscraper into a thimble."
When I first started in big data at IBM years ago, I was terrified. What terrified me was, conceptually, how to handle and surface seemingly insurmountable big data in a meaningful and coherent way for a user. I'll fully admit, for the first week, I just stared and blinked a lot. I waffled between a brain BSOD and I/O error code.
And then something wonderful happened. . . . I took a deep breath and dove in to the data sea and never looked back. It started to make sense! You never start with the data, no matter how many people try to fill your head about back-end services and data mining, you start with the USER. That is not to say everything is important at different times. It absolutely is, but you have to put yourself into the user role.
As a user, "What am I ultimately trying to accomplish?" but be careful, at this stage, never address the HOW, only the WHAT. The 'How' will come later, and is it's own crucible.
Some answers I have gleaned in the past:
"I need to be able to detect possible fraud, and fraud patterns and pass them on to an investigator within x minutes."
"I need to create an intuitive workflow and implement it asap."
"I need to validate batch-scanned OCR documents and place them into an intuitive structure for discovery and workflow tasks."
If you notice, there is no 'How'. "I need a button here to..."is not our goal. Once you divorce the 'How' from the 'What' you will start to begin to innovate. Focusing solely on the 'What' will, I guarantee, get you to an innovative solution.
Here is the simple process I have used, even before the rise of Design Thinking: (all answers come from users, with the exception of 'How', though the design validation does come from users)
Who: Who are your users?
What: What is the user trying to accomplish?
Why: Why is the user trying to accomplish these tasks?
When: When does the user need to see what?
How: How will the user accomplish these tasks?
WOW: What is the WOW factor for your specific users?
Bear in mind that I am not saying don't do "As is/To be" at a very high level. Assessing the user's current technological ecosystem and day-to-day is really key to understanding pain points, the 'As is', and redefining the 'To be', agnostic of the current model. What I am saying is understanding the 'What' and the 'Why' in tandem. If you do it right, you can frequently redefine the entire experience, at a high level, to be something so refreshing and efficient that your users will be astounded. Doing a better 1-off is not an acceptable solution if you have the resources to redefine and implement it properly in an Agile/Lean, highly iterative, process. If you are an SSDD Designer, you have lost your passion.
Ok, so if you understand that, next is to really assess what needs to be surfaced and, more importantly, WHEN it needs to be surfaced. Those of us in enterprise software have seen hundreds of screens of very dense data, nested grids, and with far too many controls, on one screen. Yes there are distinct personality types that do prefer dense data screens, but, BUT, if we do this right, you can satisfy both sides. Notice there is still no 'How'.
Once you diagram out "What' and 'When', you can now focus on the 'How'. While you may have touched on innovation in the last paragraph, here is where we really can make innovative strides. Be very careful to continuously go back and re-validate the 'What' and 'When' as you now start the 'How'. Frequently you can go down a rat hole and realize that there were misconceptions in the first 2 phases that make phase 3 invalid. This phase is incredibly iterative and requires much user validation as you move through the development of the product.
This brings us to the WOW. The 'Wow' will surface throughout the entire process, in different manifestations. Never discount 'Blue sky' comments as you go through this process. The "What if....", "It would be awesome if....", "In a perfect world...." are very key phrases. It gives you some insight into potential iconic outcomes that will empower and delight your users.
Don't give up, regardless if you are designing a small software app, web app, or honking big data enterprise software product. This works if you work it.
More deep dives into this process in the future.
I was so stressed on the SATs that, briefly, I forgot how to spell 'the'. . . . .true story.
