Python Moves to Dismiss GIL and Grow Concurrency

The removal of GIL removes a major obstacle to multithreading, making Python a multicore language.

The Python board of directors intends to make a proposal to mark the culmination of many attempts over the years to remove the Global Interlock (GIL). This will remove a major roadblock to multithreading, making Python a multicore language and significantly improving its performance for workloads that benefit from parallelism.

This way, first-class support for multithreading and concurrency is one step closer to becoming a reality.

Why remove GIL?

Python’s memory management system keeps track of object usage by counting the number of references to each object. When the count drops to zero, the object is scheduled for deletion.

This reference counting mechanism is not thread-safe because Python was created when multiprocessor systems were rare and multicore processors did not exist. Instead, Python achieves thread safety by allowing only one thread to access an object at a time. This is the purpose of GIL.

Many projects have tried to remove GIL. They allowed multi-threaded programs to run faster but at the cost of degrading the performance of single-threaded programs. Since most Python applications are single-threaded, this could have been a better tradeoff. Although GIL refinements have improved its handling of multi-threaded applications, it remains a serious bottleneck.

Must Read: What is Search Engine Optimization?

The core Python developers eventually decided to remove the GIL from CPython, but only if it could be done without slowing down single-threaded programs.

How Python will work without GIL

Current proposals for a GIL-less edition of Python use a combination of techniques to make reference counting thread-safe and leave the speed of single-threaded programs intact or affect it minimally.

• Skewed referral count. Counts of objects accessed by a single thread would be handled differently (and faster) than those accessed by multiple threads. Since most objects are accessed by a single thread, the impact on single-threaded programs is minimized.
• Immortalization. Some objects, like None, never need to be deallocated, so you don’t need to keep track of their reference counts.
• Thread-safe memory allocation. A new memory allocation system for CPython objects will make it easier to track objects in the garbage collector and allocate memory in a thread-safe manner.
• Deferred referral counting. Reference counts for some objects, such as top-level functions in a module, can be safely deferred. This saves time and resources.
• A revised garbage collector. The CPython garbage collector cleans up cyclic object references, where two or more objects contain references to each other. The GIL-less build makes many changes to the garbage collector, such as removing the “builds” system for tracking objects.

The biggest challenges to eliminating GIL

The biggest challenges present in this plan are not just technical, although the technical challenges are daunting. What looms even higher is how to make the rest of the Python ecosystem adjust to these changes and ensure that a GIL-less Python doesn’t create more problems than it solves.

Wouters states, “…any third-party code changes required to accommodate builds without GIL should work on builds with GIL (although backward compatibility with Python will still need to be addressed).”

The other big challenge, as mentioned above, is “bringing in the rest of the Python community,” Wouters said, “…and making sure that the changes we want to make, and the changes we want them to make, are acceptable.

“Before we commit to going full GIL free, we need to see community support,” Wouters said. “We can’t just change the default and expect the community to figure out what work they need to do to support it.”

The Python community experienced huge growth issues when transitioning from Python 2 to Python 3, so any major changes, like removing the GIL, would have to be fully backward compatible. As Wouters said, “We don’t want another Python 3 situation.”

Beyond the dangers and challenges lies a great reward: A Python that finally supports the parallelism programmers expect in the 21st century.

Must Read: Incorporating Artificial Intelligence Into Business