Debugging awscli2 test failures

I’ve been maintaining awscli2 package in Fedora for a couple of years now. It’s a large Python package with an extensive test suite that typically takes hours to run. Upstream usually releases new version every other day, and I’ve been using Packit to automate rebases in Fedora, configured so that updates happen only on major or minor version bump, to reduce the frequency a bit. It’s not unusual for a rebase to break things - usually some tests start failing, but it’s not always obvious what the root cause is.

This time the test suite was failing only on Rawhide - a quick dependency check revealed that the issue is almost certainly related to prompt-toolkit that has been rebased to the latest upstream version only in Rawhide. However, nothing obvious came up when going through code changes.

It was time to reproduce the issue locally. Since I needed Rawhide environment, I used mock to reproduce the RPM build failure and then a mock shell to access the buildroot and trigger one of the failing tests. There was no exception, no error messages, only failed assertion caused by empty output from the tested wizard application. Seeing that I recalled dealing with a similar issue in the past, but I didn’t remember the details. I tried to search in git history for any hints but came up empty. That’s what motivated me to write this blog post in the first place - to have something for future reference.

I had no choice but to dig deeper and deeper into the various abstraction layers until I finally figured out where the problem originates and realized that there actually is an exception being raised, but it is later being swallowed by one of the methods of a test helper class PromptToolkitAppRunner:

def _do_run_app(self, target, target_args, app_run_context):
    # This is the function that will be passed to our thread to
    # actually run the application
    try:
        app_run_context.return_value = target(*target_args)
    except BaseException as e:
        app_run_context.raised_exception = e

The exception is stored in app_run_context (as it turns out, to be used for assertions in some other tests where certain exceptions are expected), but not re-raised, so it doesn’t show anywhere in pytest output. Adding raise to the except clause makes the test fail with a traceback, revealing the whole story:

...
File "/usr/lib/python3.14/site-packages/prompt_toolkit/layout/containers.py", line 309, in preferred_width
  dimensions = [c.preferred_width(max_available_width) for c in self.children]
                ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.14/site-packages/prompt_toolkit/layout/containers.py", line 2629, in preferred_width
  elif self.alternative_content is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ToolbarView' object has no attribute 'alternative_content'

It turns out that the parent class of ToolbarView, ConditionalContainer from prompt-toolkit, gained a new alternative_content attribute, but ToolbarView constructor doesn’t call the parent’s constructor and thus this attribute is never created.

A simple one-line patch fixed the root cause and 44 failing tests.