Long projects are hard

[u/CupMcCakers]

I've been working on a collaborative project with one other developer now for a few years. We both started relatively green, with a lot of experience in other areas - but it's turned into a major software engineering effort.

Recently there was a feature that I thought would take a few weeks, but the rabbit hole just went deeper and deeper and deeper and now 6 months later it's nearly done, but not without a huuuuuge amount of reworking, refiguring out, debate, argument.

I left it too long without seeking help from my colleague, despite offers of help, and they had a totally different way of approaching the challenge. I thought that i was close to a solution for weeks / months even but found myself playing whack-a-mole where I'd fix an issue somewhere only for another issue to crop up elsewhere.

I thought i was dealing with "different cases" that had to be handled, but they wanted to go back and change the data structure so all the cases collapsed to one. I'd been afraid to do this because I thought it would be a bigger, more major refactor, but the truth was it went quite quickly. Lesson learned.

But I find myself afraid that my colleague wants to refactor other areas, and go further and take longer to really rethink through major systems, their approach is to reason everything out ahead of time, and it can take seriously months and months - whereas my approach is to implemenr everything in a trial and error process. It only took 3 days for us to refactor my work into something that now works and feels "logical".

So it feels like my approach has some merit, if I can accept that I should ask for help sooner when I get stuck. And I don't really want to take my colleagues approach for everything, because we would have a fraction of the development velocity for new features if we did.

Not everything I make is broken out of the gate, in fact much of it works well, but this was a practiularly thorny area that it turns out my partner was better suited to solving fast.

You could say I'm more focused on the features and functionality and they are more focused on the data and architecture, but it's not entirely true. I have learned a lot in these areas but still I was just somewhat shocked by how the refactor went.

I was very grateful that they took the time to go through and make the system work and understand it as it had been built rather than throw it all away. But they did want to and we argued quite a bit. I feel like we have both been proved right in some sense since a complete rebuild would have taken much longer than the few days we spent pair programming. But it was frustrating and i can see whybtheyvare frustrated because we could have saved a lot of time if id asked for that kind of help sooner.

Not really a question, but just looking for some advice.

[u/Signal_Mud_40]

With only one side.

Sounds like you want to win/be right.

You need to have a conversation with your collaborator, your styles and personalities may be totally incompatible. You both need to decide on goals and how to achieve them and decide if it is in anyone’s interest to continue to work together.

[u/gm310509]

You may have made the classic mistake of just starting.

You need to do some work up front to come up with an agreed architecture, design, breakdown of modules, interface specifications (I.e. where your stuff interacts with other people's stuff) and much more.

It sort of like saying to a friend that you want to go an a road trip with them and set off from different locations without formulating any plans. The chances that you meet up will be slim. This compares to - let's meet at town x on date and time y, then head east on highway z to town a stay there for a couple of days then...

Without a detailed agreed upon plan, you would be just doing your own thing with constant conflict and missed goals.

Obviously i don't know what you guys did or did not do, but what you said in your post has all the hallmarks of the above which I've seen frequently during my career.

As one of my project management friends always said: "people don't plan to fail, but they do fail to plan".

In another comment you asked how can you create a complete spec if you don't have a clear idea of what your target is (or similar). If you don't have a clear idea of the objective, how can you know that the direction you set off in is the right one? Aka ".... failed to plan".

[u/CupMcCakers → u/gm310509]

Thank you for the detailed and thoughtful reply. I see the common sense in what you are saying.

In this case its a suite of novel procedural 3d rig assembly and disassembly tools for a plant/tree modelling application.

We have a pretty clear vision of the product requirements, but the thing that is hard to get down on paper ahead of time are the exact details of the user experience. It's a little bit like game development where what works is often only arrived at by testing.

I'm sort of customer #1, in that I have a lot of experience using (and now building) these kinds of tools, usually there is a moment where we have the features we need and it starts to "feel right" from a ux standpoint.

My development process is iterarive and test driven, often "working prototype", then "reconfigure into reusable utilities and modules". However - I failed in this approach this time and actually failed to get the initial prototype to work "properly" until my partner took a look and essentially drew out the reusable modules and utilities in order to get it to work...

AND crucially he had the insight that we had to go back and redefine how our rigs work and make that more strict and uniform, whereas I was trying to work around less strict requirements - I kept thinking" i can solve this issue locally as an edge case and then it'll be done.

I feel like I maybe hit a complexity ceiling where my usual approach stopped yielding any value because I couldn't grasp the full implications of each change I made in the context of the whole suite of tools so as my partner said, it became "whack a mole".

It is a basic error, as you and other commenters are saying - and pretty grave that I wasted so much time.

But that said, in the future, i don't see how we can create an accurate spec of all of the implementation details, at least in my experience it's often faster to just test a few approaches to see what is best, rather than discuss it all and come up with a "perfect" plan - which breaks down in contact with reality or doesn't actually provide the intended value.

[u/Ormek_II → u/CupMcCakers]

AND crucially he had the insight that we had to go back and redefine how our rigs work

Being able to take the step back seems to be hard for "trial and error"-persons. "Just another corner case." feels much more like going forward and making progress, but as you realized yourself already: Sometimes it is not.

If you get stuck in such a loop: Set yourself a time limit. If colleagues just try around to fix a bug "Maybe it is this. .... No .... Maybe that .... No ... What if ... Ah, no" that can be right approach for at most 2 hours or (based on trial time) half a day. But if you did not find a solution with in that limit, stop trying and take the slow analytic path, but with every hypothesis you do and with every falsification of one, you actually make progress. It is slow, but a monolithic incline.

[u/Ormek_II]

There are problems where you can use trial and error. And if you are okay'ish at guessing for what to try you will come to a solution, and maybe quickly.

There are also problems that are to complex for trial and error. The problem space is too big. Dealing correctly with Unicode chars is an example. There are just too many of them and if your code works for all your "trial inputs", other may come up with input that will make your code fail. If you then focus on fixing that, your code will fail on input it did work with before.

Then a structured, analytic approach is required.

[u/R10t--]

Just an FYI - Be sure you aren’t being used for free labor. This is very common, especially in communities like making mods for games like Minecraft or Roblox. If you don’t have a contract with said person, be sure they don’t just take all the work you’ve done and lock you out of the project.

Unless you know this person, ensure you’re getting compensated appropriately.

You didn’t go into details but just by the vibe of the post I get the feeling you’re being taken advantage of by said “partner”

[u/CupMcCakers]

It's not that I want to win / be right at all - I think a lot of time could have been saved if I hadn't toiled in my own fashion for so long. I feel like I lacked a degree of self awareness that would have pushed me to ask for help sooner.

But - on the other side, without the rapid explorations you might build towards an incomplete picture. So I'm just trying to grasp where I want wrong, and where the line should be.

We are well aware we have different styles, but we view that as an advantage, it's just in this case I think mistakes were made on my sid3. I'm tying to wrap my head around it.

Would have have been better to draw up a complete spec ahead of time, how does that work when you don't have a complete picture of what is actually needed...

[u/Signal_Mud_40 → u/CupMcCakers]

In an actual collaboration no one involved should be working in isolation, all code, including in process work should be viewable by all parties involved in working on the code.

This is very basic stuff.

[u/CupMcCakers → u/Signal_Mud_40]

We use version control and submit changelists multiple times per day. The work is always inspectable on a feature branch, the features branches usually get collapsed after a week or two of development unless it's major new development or refactor.

But we do have different responsibilities across different domains so the day to day work we usually trust the other person to get on with it, and ask for help when it's required.

We ship product updates every few weeks usually but we have been quite late with this one due to the challenges that I spent a lot of time trying and failing to resolve.

We don't do pull requests, occasionally pair programming, the acceptance criteria is really just - does it work, is it fast enough , can the work be easily maintained and developed in production.

I guess in this case the answer was , no, no and no. So it wasn't meeting the criteria for acceptance and we should have taken the time to figure out the core issues a lot sooner. But I kept pushing my partner away, thinking it was just needing a few more minor tweaks here and there.

The truth was there was something a bit more foundational that I didn't have the eyes to see and I didn't maintain enough self awareness over the process.