Startups and Partnerships

Every startup I worked with was keen on strategic/channel partnerships in its initial time. These partnerships happen where product or service is complimentary and has great possibility of going together. For the startup, the idea behind partnering with established organization is to have access to their customer base and sales force. For the large organization, primarily it is the breadth of products/services they offer to their customers and some customer acquisition as a bonus.

Image Courtesy: SPRING Singapore

This looks like win-win for everyone and that's why so many strategic/channel partnership deals get signed.

However, I have never seen any customer acquisition happening in reality for the startup. Initially, when my primary role was on tech, I used to ignore these partnerships. By the way I am not the only one saying this, Paul Graham also says in his article:

"Partnerships too usually don't work. They don't work for startups in general, but they especially don't work as a way to get growth started. It's a common mistake among inexperienced founders to believe that a partnership with a big company will be their big break. Six months later they're all saying the same thing: that was way more work than we expected, and we ended up getting practically nothing out of it."

After moving to the business side, I started talking to partners myself and at the same time started investigating why things don't work and how to make them work.

When following up with the partner leadership, I heard the constant answer from everyone, "We shall let you know when any of our customer asks for your kind of product."

Later, a friendly sales person told me his point of view:

"Bringing money from anyone's pocket is a tedious job. So we (the sales people) sell only the items we are most comfortable with and what gives us most benefit meeting our numbers." 

Any new product doesn't fit into this criterion. Especially when it comes from a partner startup because customers also don't feel very comfortable with a new product and the original deal may also get derailed.

So! partnerships don't work.
But approaching every small customer with all sales effort is a tedious task too.
Look around, you will see so many partnerships working. What they did to make it work? Is this just the brand name that works? Or there is some solution in this chicken-and-egg situation.

Paul Graham also says in his article that scaling is a good-to-have problem. But till you reach there, founders has to sell themselves. They have to do it manually in a way that doesn't scale and they have to delight the customers. That's the only way to acquire initial customers.
To add to that I have seen that even professional sales people fail to sell a new product during initial times. Founders and other entrepreneurs of the team have to do it.

The trick for making partnerships work came from another friend who is a sales person himself and developed/managed partners for his company.
According to him, you need to sell and give first customer to your partner rather then expecting them to bring first customer to you. When they (partner) see new money (and customer relationship) coming without making sales effort, they start taking interest as they need to service the customer, keep him happy and get more orders from him.

Image Courtesy: RoI Investing

In this process their teams get more familiar with your product and establish a process for repeatable business. You may need to do a lot of effort in this process for the first time (and may be couple more times) but when the channel is setup, it really works best.


According to him, it takes about 2 years for fully nurturing a channel partner.
Though the trick looks fully logical to me and my friend has earned success with this, I am yet to try this. Will post my experience further.

Understanding AWS Shared Responsibility Model for Security

I heard many people saying that they need not worry about security of their application (or it is  automatically PCI compliant) just because the application is hosted in AWS EC2.

This is a big misconception. AWS has good literature about security and it clearly mentions shared security responsibility model.

The presentation shared here was presented in the AWS meetup in Bangalore organized by Jeevan and Habeeb. As purpose of this was just to make people aware of their responsibility, details of any topic are not covered here. Also this does not talks about AWS services in detail.

During the presentation, people wanted to understand details of some AWS services and also wanted to deep dive into each aspect. Many of the questions about AWS were discussed in detail by Shailesh who is architect at AWS and was present in the meetup. For other longer portions, we decided to organize follow up meetup focused on each topic.

Shared Security Responsibility Model of AWS from Akshay Mathur

Challenges with Application Visibility

After I create a web application and make it generally available to people for use, the first and foremost challenge that comes in front of me is to find out number of servers to be deployed for sustaining the incoming traffic. For determining number of servers I need to know the traffic that is landing on my servers. Knowing aggregated numbers does not help here really because the traffic does not come at a constant rate. Understanding the traffic pattern is interesting. To start with I need to provision servers based on maximum load unless I have a solution that automatically scale my servers based on traffic.

Once I make sure that all the traffic is being taken care properly, I start worrying about the user’s experience. The very first experience builds based on how much time users need to wait after a click to the page load. I majorly divide it into two parts: one is the response time from server and other is the time taken by browser in rendering.

When everything looks good at gross level, time comes to drill down deeper and figure out what areas (URLs) of the applications are being used more than the others. Typically these are the critical parts of application workflow and require special attention for availability as well as response time. When I see that traffic to some URL is going very high, I consider dedicating servers for that traffic and try optimizing that code path.

When high priority infrastructure items are taken care of, I would like to know more about users. First point is how many users are visiting, they are new or returning, after how much time they return, how many pages they look or better what pages they visit in what order, what browsers or devices they use, from what they come, how they are distributed geographically etc.

Next I want to figure out how much of the traffic is coming from non-useful bots so that I can block that traffic somehow and focus on the real traffic. I would also like to detect (as well as prevent) traffic sent with the malicious intent by an attacker. The malicious intent may be DDoS kind of attach for impacting availability of my application or a try to steal data or just an attempt to harm the user interface.

When I come up with new version of my application, I would like to compare its performance with the version currently running. Only if I find the performance acceptable, I would like to expose the new version to all my customers.

Giving the high level visibility into these areas is not enough. When something goes wrong, the tool providing me application visibility should also help me debugging the issue and allow me to drill down to the individual request level.

Google Analytics is generally known to fulfil most of application visibility aspects. However, that does not satisfy the need because of following reasons:

• Google Analytics only capture the traffic in which HTML page is accessed. All my ajax calls and requests to other resources are not included
• Google Analytics gets triggers only when JavaScript on the page is executed. Access of the page by bots in not included

So what I see in Google Analytics is the subset of the traffic hitting my servers. When reports say that the bot traffic amounts to more than 30%, I can’t accurately plan based on Google Analytics number. Also Google Analytics being a client side JavaScript tool has no understanding of my deployment infrastructure and neither helps me drill down for issues not help me scale.

A piece that is in traffic path is load balancer. But load balancers like AWS ELB, NginX or HA Proxy provide very limited visibility into traffic and are not really helpful from insights point of view.

[The post has also been published at Appcito Blog]

Techniques for Scaling Application with Security and Visibility in Cloud

When the traffic to your web application serves increase, you need to do a better job than just horizontal or vertical increase in servers.

You not only need to bring the mind shift on the software architecture side and make the application scale-able at various layers but also deal with the challenges with infrastructure optimization.

In the TechNext meetup Pune, the topic was discussed at length carefully observing the challenges, requirements and possible solutions at every stage of growth.

The discussion was around following topics:

• Architectural mind-shift for scale
• Infrastructure challenges with traffic growth
• Load balancing
• Content switching
• Content optimization
• Traffic insights
• Security against data theft
• Protection against BOTs and traffic surge
• DDoS attack prevention
• Continuous delivery

Techniques for scaling application with security and visibility in cloud from Akshay Mathur

Social Recommandation that Does NOT works

A few days back a family friend couple came to me with the proposal of exploring a "Business Opportunity using Social Network". Quickly I figured out that they are also telling me the Amway story of Multi Level Marketing (MLM) business.

Image Courtesy: First Class MLM

I am hearing Amway's presentations for about 15 years now and have seen them changing over the time. Typically, the presenter tries hard to convert me into another member in the chain and as soon as s/he feels not happening so s/he leaves.
I have already told this couple that I am not interested in MLM businesses so this couple was just doing their practice with me rather than insisting me to purchase. So, we started discussing more on why there is poor adoption of Amway products.

My wife's first concern was that the products are generally very costly as compare to the products available in the local market. 

They told that Amway says because of higher price point local shopkeepers are not able to keep the products in shops and they are sold only through the agents.

They also told that the products only look costly while actually they are not because they come in very high concentration and are directed  to be used after a lot of dilution.

Then they started sharing their own experience of the products they are using.

Both of the families had to take care of the kids and other household stuff so we stopped the discussion at some point, but I thought a bit more on two things they said:

• MLM sales channel is better than the traditional channel of distributers and retailers because the consumer is directly purchasing from the manufacturer. In this way company saves the commission of the retail chain and shares the saving with the consumers (i.e. the agents).
• We anyways recommend products to our friends. In this case we get some incentive on the recommendation.

My mind got puzzled because the social recommendation theory is actually correct. We are following the same in our company ShopSocially and it is giving us good results. This has been proved in multiple researches including the latest Forrester research that the product recommendations by friends and family are most trusted. Data at ShopSocially also suggests that the revenue of all our partner merchants increase after their customers start recommending the products via social media channels.

Social Commerce Strategy : stuff that really works

Then the big Question is why social recommendation works adversely in case of companies like Amway. Why people start creating distance from the friends recommending Amway products.

Why?

As soon as incentives come into picture, trust goes down dramatically. Amway products are appreciated only by the people in Amway network. These people get direct incentive when someone purchases any product because of their recommendation. In case of traditional channel, the company or shopkeeper does not offer any incentive for recommending a product so such recommendations are much more trusted.

Image Courtesy: The Telegraph

Additionally, in case of traditional channel, the distributer and retailer take risk of purchasing a product on behalf of customer and this gives a very comfortable feeling. Another big comfort that typically all local shopkeeper provide for minimizing the risk of customers is by allowing to purchase small quantities for trial and no money-back guaranty can beat that.

Still there was a small question, ShopSocially's partner merchants also incentives their customers for sharing the feedback and comments about the products, then why the recommendations still trusted?

I got the answer when I was able to differentiate between incentivized sharing and incentivized recommendation. In when incentive comes to user just on sharing, the user is still free to write the views about the product. It does not matter if the views are positive or negative.

But if the incentive comes only when someone buys on your recommendation, you have to write positive. It does not matter that you really like the product or not. This is why everyone connected to a MLM company speaks the same language. And this exactly why such social recommendations, even by well connected people, are not trusted even after 50 years.

Celebrating the bugs in code

What rubbish? How could you celebrate bugs in the code?

But I do! and there 2 aspects or ways to this celebration and you should do both.

Image: World Creativity and Innovation 

1. Party to the team:

In fact many organizations have this unwritten rule: "Whosoever breaks the build or breaks the product in field (especially at night), throws a party for the team".

This rule helps a lot keeping the quality high even without having a dedicated testing team. Whenever, we had stopped enforcing this rule, we observed more issues from the field.

In the party forced on the person whose code becomes the culprit, people talk about the bug and the worse situation raised because of the bug. As no one wants to be in that situation, people put more focus on what they are committing and test the changes better resulting into good quality product.

2. Learning debugging:

Debugging is an art that every developer should learn. Bugs from field are the only way for learning efficient debugging with limited resources in very stressful situations. They also make you think how to fix the issues in least possible time with minimum impact. Staying calm and planning your steps methodically is the key of success in the unfortunate situation.

Image: DaVinci’s Classroom Blog

Image: Surprise Industries

Releasing Software without Testing Team

This article is prepared for presenting in Software Systems Best Practice Conference, Anahiem, California. Final version is available on slideshare.

Image Courtsy: softheme

We can’t imagine software development without testing. Programmers are human and they’re bound to make mistakes. We use testing as a tool to ensure that defects are caught before we ship.

Good software engineering practice requires us to isolate development and testing efforts. This is commonly implemented by assigning each of these responsibilities to different teams. Along these lines, the idea of the magical ratio of testers to developers was conceived.

In order to improve quality, we intuitively increase the ratio of testers to developers. As a direct consequence of increasing the testing effort in this way, budgets and time to market are directly affected. There is also a hidden cost at work that results from the interpersonal issues when two teams with different objectives try to work together.

Development teams that are directly responsible for the quality of the code they ship reveal several desirable advantages. By internalizing quality goals, developers are forced to adopt a ‘test first’ attitude. This approach allows them to get better over time at managing the expected standards required to ship software to customers. By eliminating the overhead of inter-team co-ordination, single teams find agility and are more responsive. Turnaround time for bug resolution is also reduced to it’s minimum. These are great qualities to achieve.

Software Development Life Cycle

Before we can review the role of the software development team, we will need to revisit the traditional software development waterfall model and the role the team plays in this model. In this model requirements analysis, system and technical design, development, testing and release are each discrete steps. In practice, management and development teams rarely have the freedom to implement this model in this way. Let’s take a closer look at what the underlying forces are that disrupt the simple flow of this model. 

Requirements

If we think of any software project to be governed by technical and non-technical requirements, then the key underlying constraints are time and money.

At the start of the cycle, we describe the functional requirements or what the software must deliver. These are created on basis of our current knowledge of what our customers want. As market demands change, functional requirements must follow. In many cases, customers are themselves not aware of what it is they want.

Further, written functional requirements are at best an abstraction of what is really desired. It is left to development team to fill in the details and plug the gaps. Thus, after having created the functional requirements, we are still left with lingering questions. How do we know for sure that what the development team delivers will meet the requirements? Can all the requirements be met within the budget defined?

Considering these factors, we allow ourselves to revisit and change the requirements.

System Design

Since requirements are a prerequisite to the design stage, many of the challenges we found in the first stage also affect this stage. You could require that designs eliminate the need to revisit development even if the requirements were to change. However, this approach risks over-engineering the software and increases the development complexity. It will result in significant waste when major changes are made to the original requirements in order to keep pace with the market.

Development and Testing

Even with the backdrop of the flawed waterfall model, the implicit expectation is that the software delivered has to be of excellent quality. In a nutshell, it must meet the original every-changing requirements, work in common and extreme scenarios that can be imagined, or at least automatically degrade gracefully in the remaining scenarios that could not be feasibly accounted for or imagined in the first place.

Delivering this magical software is the joint responsibility of development and testing team. As we’re already aware, joint responsibilities lead to imperfect outcomes. Thus optimizing the tradeoff between our expectations and the ability for this joint team to deliver is of great importance to delivering great software.

Let's dig a little deeper and see how development and testing teams work. For the sake of simplicity, we will assume that testing is given the same importance as development. For instance, requirements are discussed with both development and testing teams, and both the teams are able to begin their work together.

The problem

Communication Issues

While development team does low level technical design and starts coding, testing team starts with writing test plan and test cases. At this point, only black-box testing stuff can be planned because actual code is not present for any white-box testing approaches.

As customer-facing teams learn more about the market and make changes in the original requirements, these changes need to be communicated to both teams, a costly and imperfect exercise. Further, as the development team makes progress and learns more about implementation limitations, requirements and designs will change to keep to delivery schedules and budget limits. In many cases, these changes remain implicit within the development team and their importance is realized only later.

Thus, the development and testing teams understanding of the software in development begins to diverge. As anyone with experience in testing will tell you, delays and imperfect communication results in unexpected pauses, rework and a great deal of frustration. Features that commonly go ‘missing’ or are found to be unexplainably ‘broken’ in the build routinely halt testing efforts.

Coordination Issues

The perspectives that the development and testing teams adopt are also at odds. Testers need to break the software whereas developers adopt an implementation perspective. As the perspectives are different, the order in which they want to approach the software and it’s sub-components will be different. However, in order to write comprehensive test cases, testing team requires implementation details of features that the development team will simply not be prepared for.

Development schedules also need to be adjusted for to help even out the workload of the testing team. As a result, both developers and testers have to compromise on their natural order of work and come up with a mid-way build plan. Both teams are now constrained by this plan and must ignore any creative optimization of the plan to suit their individual objectives. Failing to meet the build plan affects both teams and raises more coordination issues.

Mindset Issues

Duality in rewards and holding the two teams accountable also plays a significant role in further dividing the two teams. For instance, on uncovering embarrassing or severe defects – the testing team is pushed to tighten their processes. On the other hand, a successful release invites praise for the developers.  

The testing team essentially isolates developers from the how the software performs with end-users. If the two teams work in a staggered mode, developers simply throw code over the wall to the testers to test and move on to their next responsibility. As a result, developers miss out on the opportunity of processing valuable feedback from the field in context of the current release effort. Now bug resolution cycles are also longer as developers have to pre-empt their current responsibilities, switch context to fix bugs.

Isolation from the end-user also has a subtle effect on individuals of allowing overall quality to slip. Developers begin to believe that they must now write code to simply get past the test team. As someone else is responsible for delivering quality, now in the back of developer's mind, it is okay to write code that does not take care of all cases or does not deliver the complete functionality. Also, it is fine not to do the impact analysis upfront because if the new code breaks some other functionality, testing team will report that as well and that will go through the bug fix cycle.

As testers are rewarded for the non-obvious defects find before release. This encourages attachment with bug reports. If for the testing team, the gross number of bug reports accepted for fixing is the only measuring benchmark it can be disappointing for testers and the team at large to have bugs rejected by developers.

The Problem in a nutshell

In a practical software development project, the company needs to facilitate a great deal of coordination and communication infrastructure. In spite of that, a divide gets created between development and testing team members. Overall, these things not only add to the cost but also ruin environment of the workplace.

The Solution

Simplest solution to eliminate the divide is to merge the teams. However, it is observed that superficial merging does not work.

Superficial Merging

In an experiment to reduce this divide the two teams were merged into one 'Engineering Team' with single reporting lead.

This is a cosmetic change and resulted into negative impact. Processes stayed the same and since the engineering manager, being from development stream, failed to understand the issues of testers.

In a different experiment, developers were asked to alternate their primary development roles with testing responsibilities. This also failed for following reasons:

• Developers were pre-empted in the midst of their testing assignments as and when they need to address urgent issues from previous releases.
•  Some developers refused to be rotated to testing team.
•  Testing team felt that developers coming on rotation need a lot of training for adopting formal testing and were not being helpful.

True Merging

True merging requires merging the objectives and the functions of both teams. This implies that there are no specialist positions. Each individual must do both development and testing. Thus, merging teams is difficult to implement as an afterthought. The complete process right from planning, recruitment, execution to the delivery needs to be aligned with this ideal and everybody has to work with this approach.

The company management needs to make way for new processes. For example, when recruiting, the job profile needs to communicate to candidates that they are expected to wear both testing and development hats. Context switches between development and testing must be instantaneous. Interestingly, no formal testing documentation is needed – eliminating written test cases, test plans and test reports. The number and quality of bug reports also goes down.

All this looks horrible but it works in favor of quality. In the remaining document, we shall discuss that in length.

How it works

As said earlier, there is no dedicated testing team. Everybody involved in the project knows how the software works or what is expected from it. The waterfall model implementation remains the same. Just the development and testing is carried out differently from how two teams would approach it. Individuals are better aware of changing requirements and are team is co-ordination is better synchronized.

When work begins they are free to work in the sequence they prefer and make optimizations and adjustments in the schedule as needed. While estimating time for a feature, members add additional time necessary to smoke-test the overall build after integration.

Developers are skilled in basic testing techniques. For doing a better job in testing, the developer does a thorough impact analysis of any change he makes. Access and understanding of the details in code can increase the accuracy of impact analysis and including testability hooks. Knowledge of the code also helps in running better test cases and choosing better test data. In this way trivial bugs are resolved in the development environment itself.

When a developer encounters a bug in the code written by someone else, he discusses the bug with original writer and together they come up with a plan for resolution. Bug tracking system is mostly used to serve reminders and that is why the quality of bug reports is poor as per standards. Now bug reports contain the investigation and resolution notes rather than steps to reproduce test data etc.

In this system, both praise and fire come to the same person. This makes the developer completely responsible for the product quality and encourages better quality.

Limitations

While the approach described here produces good quality software, the formal documentation of test cases, test data and test reports etc. becomes unavailable.

Also there are cases where this methodology cannot directly apply or will need to be adapted.

For instance, if a large software product is now simply being maintained, the development and testing workloads are no longer evenly matched. Thus, the testing workload is significantly higher and calls for specialization.

In other cases such as mission-critical software, the cost of managing multiple teams plus all the overheads may still be significantly less than the cost of a defect going out in the field.

Another case is of software being shipped embedded on a device where no matter how you test otherwise; test cycle on actual devices is mandatory.

One more case is of WIPS (Wireless Intrusion Prevention Systems) where so many environmental factors are at play that a radio frequency isolated place is needed for testing.

Scaling

The approach described in this paper was applied successfully with small team sizes, or in cases where the batch size of features released is small. However, the idea behind the proposal is to make the developers write good code at the first place by holding them responsible for the quality and making them think about the potential failure points ahead of time.

At large scale, if the testing team can’t be completely eliminated, it should be introduced very late in the process. For instance, if the testing team comes in the picture at customer acceptance test level when for the purpose of development team, the software is considered ‘shipped’; the development team members have to upgrade themselves by all means for delivering high quality software.

Case Studies

Please see the final version at slideshare.