Disney's Cloudy Vision - Part 1

Today's Disney has the idea backwards: Disney Parks should be imagined as places where a particular character/IP would live, not create an Avengers/Frozen/Coco area.

This has been going on since Iger, and Chapek has decided to leave that as it is, and drive the attendance and "plus-ups" route.  Instead of a fantastic, immersive experience for everyone, even if the cost has to rise, there's a push to get the base experience to "you are here", and everything else is an add-on: want to get to Disney from the airport?  Pay up.  Want to get on the rides?  Pay up.  

On top of that, they want to push you to certain areas.  They cloak it in an "improved experience", but Genie(+) is just a way to move you to the most profitable locations.  I'm sure there some good (planning staff schedules), but the main controls allow for them to point you to their idea of the "best experience."  

I just want to be able to drop into Disney World to walk around, jump on a ride, or grab some fish and chips without having to plan it 6 months out.  I really enjoyed the experience of being in Disney World - the atmosphere, the smiles, the immersion into a world away from the real world.

"Industry Standard" Has No Place In Disney

Disney has a vision problem.  This pandemic has taken this notion from "what is going on here?" to "oh!  That's what I've been experiencing the last 5 years."  I'm heartbroken to think the world of magic has become the world of "industry standard".

It's been going on for longer, but 2018 was the first hint of what was going on.  From the Disney Tourist Blog on March 14, 2018 (emphasis mine):

"Walt Disney World has indicated that this change is to bring their Florida hotels more in line with industry standards."

Disney is supposed to be where the Impossible is possible; an industry all its own without match or "standard". Other companies and people should be looking to Disney for what they can do.

From there, you can see that they've dropped the any pretense of "doing the impossible," and pivoted to "year-over-year growth for the shareholders".  

I know the "shareholders" issue is not unique to Disney, but Disney had a unique position for decades: Freedom to tinker; money as a tool, not the goal.  That freedom included allowing Imagineers to reach outside their role and find the keys that could bring the impossible to life. 

Journey into Imagination is not my favorite ride, but there's truth to what Figment says: "Imagination works best when it's set free."   

Money is needed, but Roy and Walt had a good flow going after their initial setbacks.  It may have been a slow climb, but they were trending in the right direction, and they did not have to force it.

Focusing only on profit growth has become a way to force guests to increase Disney profits, without providing anything novel, and, as you've seen recently, to take away without replacing perks that have made the Disney experience compelling. 

In a future post (posts?), I'll provide some additional details from previous trips.  It's a wild ride that even Mr. Toad would not like. 

Breath - The missing manuals

 I was not surprised to learn that I was breathing wrong, but it still feels wrong that I find myself having to fix it.  

I'll say it up front: breathing through my nose only when exercising (running, calisthenics, jump roping, etc...) has improved my post-workout recovery.  I no longer have to suck down water like I just traveled through the desert; I no longer have exercise-induced asthma from running in the cold.  

Also: breathe it all out.  You may feel like you're gasping for air, but I've found that exhaling more has improved the intake.  Try it next time.  When you're breathing heavy, do you actually exhale all the air you can, or do you continuously inhale before the full exhale, and just continue to push down the old air in your lungs.

Breath by James Nestor was a bit dry, but enlightening.  I say dry mainly because there was no real passion focused on any techniques - or the change in breathing itself.  It was, for all intents and purposes, an overview of Breathing techniques around the world and through history.  But that is what I needed.  He did the time, and it is fully worth the price.

Now, Mr. Nestor has introduced me to a host of new practices which I hope to visit over the next few months.  While I would highly recommend reading the book, his list of breathing videos on his site are valuable on their own.

One thing I did realize after reading it was just how little follow through I tend to have.  I stumbled on Wim Hof years ago, and made feeble attempts to do the breathing exercises.  They were interesting, but I did not take them seriously.  I'm still not doing them, but I will revisit it after some more time with the "through the nose only" exercises (note: these are not formal exercises in the book, just some takeaways that I found).

Let's Shelf "Technical Debt" and talk about "Design Debt"

As the Agile philosophy picked up steam (and started generating consulting profits), us developers were introduced to the concept of "Technical Debt."

Technical Debt was intended as a way to present sub-optimal technical implementations in the business "friendly" concept of financial debt.  The delta between the optimal solution and the sub-optimal solution is the debt you burden yourself with immediately.  This debt gains interest as long as you do not pay it back by implementing the (now possibly different) optimal solution.

While this metaphor is somewhat useful when discussing the impact of a decision with management, since it never reaches the stakeholder report, there's (in my experience) very little traction to fully pay back the debt.  You sometime make a "good faith" payment (interest only?), but you're usually just left with something "less bad."

We need to move past "Technical" Debt.  The impact is just not there, and very few engineers (or managers) will champion the payment of something that "just works as it is."  Paying customers pay for new things, or fixing their problems.

I propose we start teasing out the truth of the debt.  One of these is "Design Debt."  Design Debt is a high stakes debt that will crush your pace, and can make a seasoned engineer cry.

Design Debt is the sub-optimal design decision made because you are using an Agile Methodology: requirements firm up over multiple iterations, so the design decision you made in an earlier cycle (sprint, Epic, etc...) is already sub-optimal.

Take, for example, deciding that you should implement an Entity-Attribute-Value data model since you heard the customer make a high level request for a system that you can "dynamically add fields to." 

If you start moving forward with that (because we're "sprinting"), prior to engaging the customer in a deeper discussion, Congratulations!  You've got "Design Debt".

And 3 years later, after you're finally going to production, and find the system cannot show a significant portion of your data because it times out querying for the data, you now have to allocate a full time senior developers time for, oh, 8 months to completely tear it out and put in a normalized data model.

If you get just one thing from the above, make it that Entity-Attribute-Value is a last resort model! 

That Was Quick

I dropped the idea of a streaming service changing the video you're watching for each view, and then AT&T gets called out for deciding to change your stream to better target ads.  I recommend you read the Verge article.

The plan AT&T lays out is much more disturbing: they'll combine data from all their properties - cellular, phone, media  change the stream - which will allow them to track you from watching the ad, to actually acting on the ad.  The Verge describes a customer who watches a car ad, and then is tracked going to the dealership. 

I suppose the positive is that you will not have to fill out that survey question about "Where did you hear about us?"

Delete My Data AND The Extracted Information

While I am generally not pro-active in insuring my privacy (I'm quite beholden to Google services), the idea of a 3rd party deleting my data, yet keeping the information their algorithms extracted from it bothers me.

How would I prove the results they generated were wrong?  What if the results have been collected over months or years, and now the Credit Agency see's me as a high risk?  The data used as input to those algorithms is gone now (yay Privacy & Right to Be Forgotten).  How are you going to prove that it was an error and should be removed?  I'm not even sure what data they were collecting.

Moving forward, what if governments worked this way: your data is only on our systems momentarily, and then removed.  Behind the scenes, you've been tagged as a high risk because of an error in the algorithm, and now you are no longer able to purchase a plane ticket.

How will you correct that situation?

It would be like an e-voting system that "recorded" the vote and then destroyed the actual input/ballot.  It flipped every 3rd or 4th vote, but there's no feasible way to prove and fix it.

So, either keep both my data that you used as input and the information you generated, or keep neither.

Did your Streaming Movie Change?

While I enjoy the convenience of digital streaming - be it Movies or Music - I am struck by the potential for some unknown hand to subtly change the scenes, sounds, dialog/vocals, or even "adjust the message".

Since I've purchased a digital copy only, I would not even be able to check to see if something did change.  Almost like a digital "gaslighting." 

Even if there is a physical, unmodified copy available, very few would be motivated to compare.  If it was truly subtle, it may be difficult to notice it.

I feel that this has been done recently - to a syndicated show in order to change the advertiser - and there being at least some discussion about it.  Unfortunately, I am not able to easily dig up anything on it.

PostgreSQL JDBC + SSL

This serves as a note to myself (and anyone else) on a simple way to connect to a SSL-enable PostgreSQL (9.4+) server using the standard java SSL properties:

Add sslfactory=org.postgresql.ssl.DefaultJavaSSLFactory to your connection string, and configure your JVM javax.net.ssl.* properties as you would for the majority of java applications.

E.g.:

jdbc:postgresql://dbhost:5432/testdb?ssl=true&sslmode=verify-ca&sslfactory=org.postgresql.ssl.DefaultJavaSSLFactory

and, if you use PKCS12 for certificate storage:

java -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.keyStore=/path/to/cert.p12 -Djavax.net.ssl.keyStorePassword=notapassword -Djavax.net.ssl.keyAlias=mycert -Djavax.net.ssl.trustStore=/path/to/truststore.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=notapassword

Re-framing Privacy

I really enjoy shows that guide me through various points of history, digging deeper into the day to day minutiae that your history classes in high school and college did not - and generally could not - show us.

You can also find nuggets of knowledge that can expand your understanding of the modern day.  This happened recently while watching Lucy Worsley's "History Of The Home" series - I think it was the "Bedroom" episode.

While discussing the lack of intimacy in the home, she said - and I'm paraphrasing - that privacy was the ability to choose who you share yourself with.  While obvious, and probably not uncommon, the quote rung in my head, echoing through the chambers.  Quick aside: it's worth watching that series just to hear about the origins of "making the bed" and "sleep tight".

It clarified my mistaken presumption of privacy as a passive "something" that you had; it was, in actuality, an action that was controlled by you.  It's what you do, not what you have.

Loss, or invasion, of our privacy, then, is the wrong way to think about the privacy problems we face today in technology, government, and society.   When I hear about it, privacy is presented as a secondary privilege, as if it were my home.

This leads me to believe that privacy should be compared to Free Speech.  In fact, privacy seems to be involved in exercising free speech: I choose to whom, of what, and how much of it, I speak.  Being stripped of privacy prevents me from effectively exercising my free speech rights.  Now the encroacher has not just read your journal, but has identified your personal expressions ("oh.  I see you like to dance as you get into the shower").

In this way, we need to think of this as being stripped of a freedom, rather than a loss or invasion.   Privacy is a choice of what you share, and how much of it you share: you choose when to stop sharing.

Unfortunately, the platforms we use are actively getting in the way of us exercising our privacy: "Look at what is going on out there.  Just take a peek.  Don't you want to say something about it.  Perhaps do something - we can help you do that something."  Kind of like having a kid around: you don't focus on their presence, you don't sense any danger of them being around while your acting out your day, then BAM!  Your kid just told your friend what you bought them for their birthday - or, worse, tells your girlfriend that you bought her an engagement ring (https://money.cnn.com/galleries/2010/technology/1012/gallery.5_data_breaches/3.html). 

Here's a thought: when you go into a store, you have a couple areas where you could exercise privacy (e.g. the bathroom).  Where can you do that on Facebook?  On any Google property?  Or even Amazon or Apple?  Every move you make goes directly into their internal data set, which they can parse whenever they want (in private too!)

This is definitely a topic to revisit soon.

Agile Development: The Psychological Toll

I have a new hypothesis about Agile Software Development: it will cause a form of "Stockholm Syndrome" along with a healthy serving of Cognitive Dissonance in a significant number of developers.  Many of those developers become ardent evangelists that will not even consider alternatives - or even deviations - from their Agile Methodology.

I've been working on or with teams using Agile methodologies for over a decade.  It will be around for a while.  Yet, there has always been a problem with those teams...there was always a feeling of tension at the end of the sprint, even though it was not acted upon by the developers; it was internalized as shame, resentment, self-loathing, and even a mild depression. 

In the last 2-3 months, some on my team, and myself have waddled through such feelings.  I'm fortunate to have been able to work through them, and to emerge with a bit more understanding of why I've had such a bias against Agile.

With a team using Agile Methodologies, there is an overwhelming urgency to complete your selected story/task/bug/issue within the sprint cycle - "that's what the mantra's say!  That's how Agile works!  Everyone else who used Agile does it, and does it better than me/us!  I cannot be the slacker!"

And then it happens. 

You've gotten a few sprints closed.  You completed your tasks in time; some were close, but it was probably just the extra cycles spent on Reddit.   Staying an extra couple hours made sure I kept velocity at the right level. 

The next sprint was not so fortuitous.  You did not finish a task.  It was only 5 story points.  I'm an idiot.  Don't worry, I'll finish it up during the first few days of the next sprint, and still complete my typical number of points.   

I'll just work a few extra (unpaid or unreported) hours.

Now you've moved into the mindset of "keeping up with the team," rather than "solve a problem effectively for this domain/customer".

We're coming up on the end of the next sprint.  I've done a decent amount of work creating and coding a solution...but it feels wrong.  I could really use a day to review and maybe (no...no!) rework the solution.   

Nope.  No reworking.  What I have implements the fix/feature.  It'll fly through testing. 

But there's something wrong with it...I really need to know...
Ugh.  Close the ticket.  Now ignore it.

There it is.  The start - or perhaps relapse - into a state of Cognitive Dissonance: You believe deeply that you are an excellent Software Engineer; you create solutions for problems that exceed and amaze not only your customers, but your team and even yourself.  Yet you have just committed a solution to the baseline that you feel is...wrong...insufficient for the actual problem as a whole.

The show must go on, so you have to convince yourself that it must be the correct solution.  Agile is good, right.  It knows something you don't.  Master it and you will become the master.   

And if others follow it, they will see me as a master....  If others follow it, I am not wrong.

Security through technology?

Rambling post ahead...just an attempt to keep writing, and perhaps generate some focused topics for later posts.

I've noticed a rapid push for more and more security tools to be installed on everything from a toaster (e.g. IoT devices) to desktops, to VMs and physical servers.

This one monitors your activity (all of it?  Specific actions?); this one verifies that blob of bits is benign (or, really, not known to be malicious); this one prevents you from using parts of the system (USB ports, CD/DVD drive).

Luckily...it leaves about 1/4th of the system available for those actual value-generating activities (hopefully those activities are ok, even if monitored).  Hopefully we all over-purchased resources so we can handle the current - and the future - security tools that will be required on our systems.

It's not unexpected.  It's a lot easier to sell a product that offers (the illusion of) control, than to be constantly vigilant, or work with those around you to improve actual security.

And it's not that these products cannot be valuable in ones goal to maintain the security, integrity, and privacy of your data, environment, and, of course, your self.  They just tend to become a way to say you've done something to improve security, without actually proving it improved your - or your customers - security.

 Then, there's the issue of parsing all this security data that the tools generate...which requires resources that also need to be secured...

The Benefit of Not Using the Team Development Environment

I'm not a "standard"-compliant type of developer.  At least, not when it comes to my development environments.

App server integrated with my IDE?  Nah.  I'll spin up a new VM with a packaging-compliant version, secured as close as possible to the deployed instance.

There's just something about creating your own deployment instance that helps me settle in.  Now I know why Tomcat and NGINX have trouble speaking SSL to each other.  

And it also uncovers trivial, yet easily hidden, deployment errors early.

Not long ago, we had an intern code up a small metrics collection component.  I was on a task that had me touching just about all parts of the code, which caused me to get quite a bit out of sync with the primary development branch.  Other developers were happily chugging along, pulling the new code, finishing tasks from the backlog.  Happy times.

I finally reached a stable point where I could start merging in the fast moving development line (trying to avoid really intense merges).  It all merged in without much trouble.  I had it building a new WAR, and felt pretty confident.

Then, I tried to run it in my (non-standard) environment.  Hmmm, what's this "ERROR" message?  Perhaps it can be ignored as an in-progress task?

"Can't create /metrics.tmp"

What?  Why is it trying to create a file in the root directory?  Time to dig!

Oh.  The new metrics code did not specify a directory - or offer a property to specify one - when creating a temporary file for collecting the metrics.  

How did it work?  Well, he was fully setup with the standard environment, which ran Tomcat from his Eclipse IDE (no, I deviate here too and use IntelliJ).  The file was created without issue since the default directory was his home directory, and Tomcat ran under his identity.

Even deploying it to a development test environment did not uncover the issue, as

The failure did not stop the overall startup process.  Everything ran, and Metrics was not a priority for testing a pre-MVP product.  It was still using a "development" environment setup, so the creation succeeded, though put the file alongside the application installation directory.

So, the next time you start following the "New Developer Environment Setup" docs, you may want to try a few deviations.  It may save some troubleshooting down the line.

JPA Follies

While I have been responsible for some - well, many - missteps when implementing a JPA (+Spring+Hibernate) layer for systems, this is one I'm pretty proudembarrased by.

TL;DR: Hibernate throws a "LazyInitializationException" within an @Transactional method, which turns out to be because I attempted to "pre-load" the caches during startup, only to have Lazy (Proxy) objects loaded and used during a subsequent (i.e. "Session closed") request.  Removing the pre-loading or using an EntityGraph to eagerly fetch the data corrected it.

I had 2 tasks to work on this week:

1. Finish the transition to Spring Data JPA
2. Minimize the latency for clients requesting data

#1 was going well, but I was still seeing > 60s "Time To First Byte" (TTFB) (#2), and that kept creeping up while adding data.  We're talking > 15,000 rows of multi-table (transformed) data being returned.

Fortunately, that high latency was only on the initial load.  Subsequent loads used the cached versions, so I was hitting < 7 second loads for 15,000+ rows of data.

"So...", I think, "why not pre-load the data!"  It makes sense, and I know there are potential pitfalls, but they should be fairly simple to work through if I search around the 'net.

Wow.  Mrs. Foot, meet Mr. Mouth.

I add in the call to load the data at startup, and then make my request, ready to praise myself for being a genius.

"LazyInitializationException - no Session" 

WTF?

I trace through the request, everything needing an @Transactional - and being called via a Spring Managed Component - is in place, and even the TransactionManager says the transaction is active right before accessing the proxied fields.

What.  The.  <Fill in favorite curse>?

I expected, and saw, the "proxy" class being used by the loaded data.  The methods extracting the fields are being called within the @Transactional (using the getter's and setter's as required; not direct field access).

It started percolating in my head that perhaps data was being loaded from another thread.  Yet, nothing else was running/processing at the time; this was the only thread loading and using data.

Well...I did have a code to load the data from the database after startup.  But...it loaded...

Oh...F...  It called the repository method, but never accessed any of the lazy loaded data, so...it cached the proxy object, closed the session, and started laughing knowing the problems that would happen (yeah, my code is a jerk).

Remove initialization code.  Rerun test.  No LazyInitializationException.  We're back to the long initial load, but that can wait for after getting everything working.

NGINX + Apache Tomcat - Certificate Proxying Adventures

I ran into a problem a while back: I identified NGINX as the best technology to reverse proxy our Apache Tomcat instance, but there was 1 particular requirement that had no real solution at the time:

- Tomcat must use X509 Client Authentication

E.g.:

Why was that a problem?  If it was Apache HTTPD, it would be easy: use proxy_ajp and off you go.  It's a well documented configuration.

But this was NGINX, and it did not have a recommended AJP bridge which would pass through the appropriate SSL headers.  HTTP was the preferred pass through.  Even more odd, I could find no actual documentation of this configuration, though it seems like it would be more common (which it might be, but only for internal Enterprise deployments, where X509 is more ubiquitous).

I could not use the backend SSL connection, since Tomcat would pull out the Proxy servers' X509 certificate for authentication, and the backend web application was not developed with proxying of client certificates in mind, so no special headers (e.g. Proxy-User-DN) configurations would work.  I could have dug into the source and added it, but 1) the work would have to go through a slew of process and vetting for something I was not technically supposed to be doing, and 2) WHY DOESN'T THIS WORK!?  IT SHOULD WORK!

Which brings us to today (well, last week actually).  

First, let's explore this problem a bit.  This will provide some insight into the why's and how's that make up the (fairly simple) solution.

What do we really want to do?  In my case, while a secure connection is desired, it did not have to traverse past the Reverse Proxy (NGINX) - the backend was secured behind private networks and other defenses, and performance was a concern.  No, we just needed to present a valid client certificate to the Tomcat server, and the Spring Security framework would do the rest.

With a direct SSL connection, easy; no thought really.  All the necessary information is there because Tomcat was performing the SSL handshake and verification, and then adding the certificate, DNs, cipher, etc... to the context.  

Okay, so just send over the Client Cert to Tomcat, right?  NGINX has a $ssl_client_certificate variable.  

proxy_set_header SSL_CLIENT_CERT $ssl_client_certificate;  

Done, and done...

Well, no, that does not work.  

One, the HTTP connector does not transfer any SSL information into your context, so the SSL_CLIENT_CERT is ignored.

Two, even once you find the magical configuration that will grab the SSL information, it fails because it does not see the NGINX $ssl_client_certificate value as a valid PEM encoded certificate.

The first part of the solution; Apache Tomcat *can* pull out the SSL context with just a simple addition to your server.xml configuration: just add the SSLValve to your Engine configuration

The information in the link provides all the necessary details for setting it up on Tomcat and HTTPD's side.  It even shows setting the SSL_CLIENT_CERT header.

But, translating that to NGINX was necessary, and not as direct as one may think.  

NGINX actually instantiates 2 SSL variables for the client cert: $ssl_client_certificate, and $ssl_client_raw_cert.  

Yet, adding those headers via proxy_set_header, and trying it with either of those client_cert variables failed when the SSLValve attempted to decode them.

Hmmm...looking at that value in NGINX, or copying it out and running it through openssl x509 produces valid results...

The problem, it turns out, was hiding both behind the scenes, and in front of my face.

Take a look at the first comment in the invoke function of the org.apache.catalina.valves.SSLValve source (line 72 in the link):

   

/* mod_header converts the '\n' into ' ' so we have to rebuild the client certificate */

And then proceeds to add back in a line feed at every space.  In this case, the SSL certificate now has duplicate ‘\n’ characters, so the format becomes invalid.

SSLValve implicitly assumes you will be using Apache HTTPD...and all the nuances that come with it.

Really, it's right out there.  Tomcat and HTTPD are both Apache, so that is where the focus is for integration.  While I don't fault them for the implementation, a bit of explicit warning is probably in order.  I may even try to update the SSLValve implementation at some point if I get time.

Still, this actually provides the final part of our solution: get NGINX to send the CLIENT_CERT the same way mod_header does.  

Oh...NGINX does not have any built-in way to modify the variable value's when assigning them to the header...

But!  It does have a LUA plugin, which comes precompiled when you use OpenResty.  Not exactly the way I had hoped it would fall out, but that's all it took.  Have LUA modify the $ssl_raw_client_cert to translate all '\n' to ' ', and SSLValve accepts it, and Spring properly authenticates the user.  The actual change required, in addition to the headers required by SSLValve, to the location section of the config is:

set_by_lua $client_cert "return ngx.var.ssl_client_raw_cert:gsub('\\n',' ')";

proxy_set_header        SSL_CLIENT_CERT         $client_cert;

Success!  Tomcat can use the client cert for authorization, and we are no longer confined to HTTPD + AJP when using SSL and reverse proxies.

There are probably other ways to fix this up, but this method appears to be a good, general solution.

The Beginning

The search for a blogging platform can be frustrating for a tech-head:

• "I want to host and manage my own!" (The control factor)
• "Maybe I'll make my own!" (Not-Invented-Here factor)
• "I want it now!" (The 'play with a new thing' factor)
• "Maybe I'll do this later..." (Fear of failing factor)

Even worse, I owned a domain (perplexedmind.com) for a good 5 years without putting it to use. It was lost immediately after my contract elapsed; I tried to get it again, but it was no longer available. Oh well. Try another variant (perplexedminds.com).

So, here I am, trying out a writing platform (quite nice so far!), and ignoring my intended goal: start a blog.

Still, I'm not too upset. The best thing about doing research, is all the tangents that you take. You stumble upon the most fascinating ideas, technologies, and, well...other things - things you may have never known existed, like "Draft," or how showering every day is causing more damage to your body than you know.

Annnnddddd...in the end, I circle back around, and click on the "Create Blog" button on Google, where about 87.6% of my online life resides (including my domain).

Now that I think about it, that "one-click. Done." deal has really started defining me. Damn. I'm one lazy guy.

Until next time...