AWS Lambda の初期化フェーズ (INIT) は「10秒」に制限されている.例えば,AWS Lambda 関数の「ベストプラクティス」を意識してハンドラ外に実装した処理が長くなったり,AWS Lambda 関数 (Python) で機械学習系のライブラリなどを多く import しようとして遅くなり,結果的に INIT のタイムアウトが出てしまうことがあったりする.
INIT_REPORT Init Duration: 10020.95 ms Phase: init Status: timeout
AWS Lambda 関数 (Python) で import の最適化をするためには「そもそもどの import が遅いのか」を確認したく,import の前後に計測用のログを仕込むこともできるけど,Python 3.7 から使える -X importtime
オプション(もしくは環境変数 PYTHONPROFILEIMPORTTIME
)が便利で AWS Lambda 関数でも使えるので紹介したいと思う💡
docs.python.org
今回は以下の「計4パターン」で試していく❗️
- AWS Lambda 関数 (ZIP) x
PYTHONPROFILEIMPORTTIME
環境変数
- AWS Lambda 関数 (ZIP) x Layer x
-X importtime
オプション
- AWS Lambda 関数 (Image) x
PYTHONPROFILEIMPORTTIME
環境変数
- AWS Lambda 関数 (Image) x
-X importtime
オプション
サンプルコード
あくまでサンプルなので意味はないけど,今回は boto3 / json / numpy / requests を import するコードを準備した👌
👾 requirements.txt
boto3
numpy
requests
👾 app.py
import boto3
import json
import numpy as np
import requests
def lambda_handler(event, context):
return {
'statusCode': 200,
'body': json.dumps({'message': 'hello world'})
}
1. AWS Lambda 関数 (ZIP) x PYTHONPROFILEIMPORTTIME
環境変数
👾 template.yaml
AWS SAM の template.yaml
を以下のように書いた.
Environment.Variables
で PYTHONPROFILEIMPORTTIME: 1
を設定している😃
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Resources:
Function:
Type: AWS::Serverless::Function
Properties:
FunctionName: importtime-env
CodeUri: src/
Handler: app.lambda_handler
Runtime: python3.12
Architectures:
- x86_64
Environment:
Variables:
PYTHONPROFILEIMPORTTIME: 1
AWS Lambda 関数 importtime-env
をデプロイして実行すると以下のように import 時間がログに出力されていた❗️(ちなみに AWS CloudWatch Logs に出力されたログは表形式が崩れてしまっていて見にくかったため,以下のログは AWS SAM の sam local invoke
コマンドの結果を載せている🙏)
import time: self [us] | cumulative | imported package
import time: 1041 | 1041 | _io
import time: 89 | 89 | marshal
import time: 826 | 826 | posix
import time: 2987 | 4941 | _frozen_importlib_external
import time: 361 | 361 | time
import time: 446 | 807 | zipimport
import time: 33 | 33 | _codecs
import time: 424 | 456 | codecs
import time: 542 | 542 | encodings.aliases
import time: 3553 | 4551 | encodings
import time: 170 | 170 | encodings.utf_8
import time: 158 | 158 | _signal
import time: 44 | 44 | _abc
import time: 427 | 471 | abc
import time: 857 | 1327 | io
import time: 82 | 82 | _stat
import time: 100 | 181 | stat
import time: 1870 | 1870 | _collections_abc
import time: 38 | 38 | genericpath
import time: 82 | 120 | posixpath
import time: 992 | 3162 | os
import time: 76 | 76 | _sitebuiltins
import time: 186 | 186 | sitecustomize
import time: 53 | 53 | usercustomize
import time: 1784 | 5259 | site
import time: 391 | 391 | types
import time: 65 | 65 | _operator
import time: 492 | 557 | operator
import time: 226 | 226 | itertools
import time: 142 | 142 | keyword
import time: 217 | 217 | reprlib
import time: 105 | 105 | _collections
import time: 739 | 1426 | collections
import time: 87 | 87 | _functools
import time: 1574 | 3087 | functools
import time: 1797 | 5829 | enum
import time: 70 | 70 | _sre
import time: 288 | 288 | re._constants
import time: 349 | 636 | re._parser
import time: 215 | 215 | re._casefix
import time: 373 | 1294 | re._compiler
import time: 165 | 165 | copyreg
import time: 1651 | 8938 | re
import time: 906 | 906 | _json
import time: 1298 | 2203 | json.scanner
import time: 1010 | 12151 | json.decoder
import time: 1093 | 1093 | json.encoder
import time: 452 | 13694 | json
import time: 142 | 142 | collections.abc
import time: 152 | 152 | token
import time: 56 | 56 | _tokenize
import time: 1129 | 1337 | tokenize
import time: 151 | 1487 | linecache
import time: 890 | 890 | textwrap
import time: 464 | 464 | contextlib
import time: 671 | 3653 | traceback
import time: 266 | 266 | warnings
import time: 621 | 621 | _weakrefset
import time: 479 | 1100 | weakref
import time: 40 | 40 | _string
import time: 569 | 609 | string
import time: 580 | 580 | threading
import time: 41 | 41 | atexit
import time: 2290 | 8536 | logging
import time: 174 | 174 | awslambdaric
import time: 309 | 309 | importlib
import time: 188 | 188 | awslambdaric.lambda_context
import time: 97 | 97 | awslambdaric.lambda_runtime_exception
import time: 1399 | 1399 | runtime_client
import time: 289 | 289 | numbers
import time: 979 | 1267 | _decimal
import time: 134 | 1401 | decimal
import time: 263 | 263 | math
import time: 131 | 131 | __future__
import time: 211 | 211 | simplejson.errors
import time: 123 | 123 | simplejson.raw_json
import time: 192 | 192 | _struct
import time: 111 | 303 | struct
import time: 128 | 128 | simplejson.compat
import time: 206 | 206 | simplejson._speedups
import time: 156 | 362 | simplejson.scanner
import time: 247 | 1040 | simplejson.decoder
import time: 542 | 542 | simplejson.encoder
import time: 368 | 2412 | simplejson
import time: 232 | 4305 | awslambdaric.lambda_runtime_marshaller
import time: 553 | 6353 | awslambdaric.lambda_runtime_client
import time: 510 | 510 | awslambdaric.lambda_runtime_log_utils
import time: 330 | 7689 | awslambdaric.bootstrap
import time: 166 | 8028 | awslambdaric.__main__
import time: 684 | 684 | concurrent
import time: 653 | 653 | concurrent.futures._base
import time: 233 | 1569 | concurrent.futures
import time: 150 | 150 | _heapq
import time: 178 | 327 | heapq
import time: 215 | 215 | _queue
import time: 306 | 847 | queue
import time: 298 | 1145 | concurrent.futures.thread
import time: 225 | 225 | errno
import time: 715 | 715 | _socket
import time: 199 | 199 | select
import time: 616 | 815 | selectors
import time: 320 | 320 | array
import time: 1888 | 3736 | socket
import time: 2242 | 2242 | botocore
import time: 1241 | 1241 | botocore.vendored
import time: 2355 | 2355 | botocore.vendored.requests.packages.urllib3.exceptions
import time: 1555 | 3910 | botocore.vendored.requests.packages.urllib3
import time: 1041 | 4950 | botocore.vendored.requests.packages
import time: 18 | 4968 | botocore.vendored.requests.packages.urllib3
import time: 24 | 4991 | botocore.vendored.requests.packages.urllib3.exceptions
import time: 1767 | 6758 | botocore.vendored.requests.exceptions
import time: 1630 | 8387 | botocore.vendored.requests
import time: 8803 | 20672 | botocore.exceptions
import time: 2310 | 22981 | boto3.exceptions
import time: 4605 | 31546 | boto3.compat
import time: 208 | 208 | copy
import time: 86 | 86 | _wmi
import time: 491 | 576 | platform
import time: 193 | 193 | _bisect
import time: 132 | 325 | bisect
import time: 189 | 189 | _random
import time: 82 | 82 | _sha512
import time: 7551 | 7551 | _hashlib
import time: 211 | 211 | _blake2
import time: 1203 | 8965 | hashlib
import time: 932 | 10490 | random
import time: 190 | 190 | _ast
import time: 1253 | 1442 | ast
import time: 160 | 160 | _opcode
import time: 548 | 708 | opcode
import time: 735 | 1443 | dis
import time: 68 | 68 | importlib.machinery
import time: 2190 | 5141 | inspect
import time: 751 | 5892 | jmespath.compat
import time: 1454 | 7346 | jmespath.exceptions
import time: 1945 | 9291 | jmespath.lexer
import time: 1121 | 1121 | jmespath.ast
import time: 2542 | 2542 | jmespath.functions
import time: 2458 | 5000 | jmespath.visitor
import time: 5000 | 30900 | jmespath.parser
import time: 1373 | 32273 | jmespath
import time: 894 | 894 | botocore.docs.bcdoc
import time: 570 | 570 | _datetime
import time: 161 | 730 | datetime
import time: 694 | 694 | http
import time: 158 | 158 | email
import time: 484 | 484 | email.errors
import time: 218 | 218 | binascii
import time: 291 | 291 | email.quoprimime
import time: 344 | 344 | base64
import time: 192 | 535 | email.base64mime
import time: 153 | 153 | quopri
import time: 105 | 257 | email.encoders
import time: 223 | 480 | email.charset
import time: 695 | 2216 | email.header
import time: 181 | 181 | urllib
import time: 1539 | 1539 | ipaddress
import time: 1116 | 2835 | urllib.parse
import time: 87 | 87 | _locale
import time: 700 | 786 | locale
import time: 1121 | 1906 | calendar
import time: 311 | 2217 | email._parseaddr
import time: 503 | 5554 | email.utils
import time: 321 | 8090 | email._policybase
import time: 464 | 9038 | email.feedparser
import time: 349 | 9544 | email.parser
import time: 205 | 205 | email._encoded_words
import time: 118 | 118 | email.iterators
import time: 444 | 766 | email.message
import time: 1176 | 1176 | _ssl
import time: 2240 | 3415 | ssl
import time: 916 | 15334 | http.client
import time: 273 | 273 | shlex
import time: 192 | 192 | importlib._abc
import time: 137 | 328 | importlib.util
import time: 6200 | 6527 | botocore.vendored.six
import time: 2665 | 2665 | dateutil._version
import time: 1312 | 3976 | dateutil
import time: 4745 | 4745 | six
import time: 45 | 45 | six.moves
import time: 2056 | 2056 | dateutil.tz._common
import time: 1089 | 1089 | dateutil.tz._factories
import time: 29 | 29 | six.moves.winreg
import time: 1994 | 2022 | dateutil.tz.win
import time: 8267 | 18223 | dateutil.tz.tz
import time: 912 | 23110 | dateutil.tz
import time: 58 | 58 | _typing
import time: 2975 | 3033 | typing
import time: 5996 | 5996 | urllib3.exceptions
import time: 1576 | 1576 | urllib3.util.timeout
import time: 3763 | 5338 | urllib3.util.connection
import time: 925 | 925 | urllib3.util.util
import time: 100 | 100 | brotlicffi
import time: 87 | 87 | brotli
import time: 89 | 89 | zstandard
import time: 2076 | 3275 | urllib3.util.request
import time: 854 | 854 | urllib3.util.response
import time: 2573 | 2573 | urllib3.util.retry
import time: 252 | 252 | hmac
import time: 8511 | 8511 | urllib3.util.url
import time: 2046 | 2046 | urllib3.util.ssltransport
import time: 2199 | 13006 | urllib3.util.ssl_
import time: 1111 | 1111 | urllib3.util.wait
import time: 1007 | 27162 | urllib3.util
import time: 30 | 27192 | urllib3.util.connection
import time: 1851 | 29043 | urllib3._base_connection
import time: 3180 | 3180 | urllib3._collections
import time: 580 | 580 | urllib3._version
import time: 78 | 78 | _winapi
import time: 102 | 102 | winreg
import time: 434 | 612 | mimetypes
import time: 1646 | 2258 | urllib3.fields
import time: 982 | 3240 | urllib3.filepost
import time: 331 | 331 | zlib
import time: 97 | 97 | brotlicffi
import time: 85 | 85 | brotli
import time: 84 | 84 | zstandard
import time: 1295 | 1295 | urllib3.util.ssl_match_hostname
import time: 3902 | 5197 | urllib3.connection
import time: 5173 | 10965 | urllib3.response
import time: 1379 | 15583 | urllib3._request_methods
import time: 619 | 619 | urllib3.util.proxy
import time: 3896 | 20097 | urllib3.connectionpool
import time: 3176 | 3176 | urllib3.poolmanager
import time: 98 | 98 | urllib3_secure_extra
import time: 2025 | 67224 | urllib3
import time: 198 | 198 | xml
import time: 219 | 416 | xml.etree
import time: 368 | 368 | xml.etree.ElementPath
import time: 543 | 543 | pyexpat
import time: 333 | 875 | _elementtree
import time: 746 | 1988 | xml.etree.ElementTree
import time: 172 | 2575 | xml.etree.cElementTree
import time: 88 | 88 | awscrt
import time: 29 | 117 | awscrt.auth
import time: 169 | 169 | _compression
import time: 327 | 495 | gzip
import time: 3166 | 119546 | botocore.compat
import time: 824 | 824 | html.entities
import time: 382 | 1206 | html
import time: 406 | 406 | _markupbase
import time: 1612 | 3223 | html.parser
import time: 2229 | 5451 | botocore.docs.bcdoc.docstringparser
import time: 2839 | 2839 | botocore.docs.bcdoc.style
import time: 2818 | 131547 | botocore.docs.bcdoc.restdoc
import time: 135 | 135 | fnmatch
import time: 85 | 85 | _winapi
import time: 79 | 79 | nt
import time: 70 | 70 | nt
import time: 82 | 82 | nt
import time: 72 | 72 | nt
import time: 73 | 73 | nt
import time: 70 | 70 | nt
import time: 131 | 659 | ntpath
import time: 716 | 1509 | pathlib
import time: 286 | 286 | _bz2
import time: 330 | 615 | bz2
import time: 378 | 378 | _lzma
import time: 478 | 855 | lzma
import time: 512 | 1982 | shutil
import time: 315 | 2296 | tempfile
import time: 175 | 175 | urllib.response
import time: 208 | 382 | urllib.error
import time: 1552 | 4229 | urllib.request
import time: 1050 | 1050 | dateutil._common
import time: 2968 | 4018 | dateutil.relativedelta
import time: 7682 | 11699 | dateutil.parser._parser
import time: 2059 | 2059 | dateutil.parser.isoparser
import time: 1534 | 15291 | dateutil.parser
import time: 3007 | 3007 | botocore.awsrequest
import time: 920 | 920 | urllib3.contrib
import time: 74 | 74 | OpenSSL
import time: 25 | 99 | OpenSSL.SSL
import time: 4066 | 5083 | urllib3.contrib.pyopenssl
import time: 359 | 359 | importlib.resources.abc
import time: 353 | 353 | importlib.resources._adapters
import time: 477 | 1188 | importlib.resources._common
import time: 223 | 223 | importlib.resources._legacy
import time: 265 | 1675 | importlib.resources
import time: 2328 | 4003 | certifi.core
import time: 1107 | 5109 | certifi
import time: 2877 | 13068 | botocore.httpsession
import time: 15381 | 52482 | botocore.utils
import time: 1084 | 53566 | botocore.docs.shape
import time: 1522 | 1522 | botocore.docs.utils
import time: 1627 | 56713 | botocore.docs.example
import time: 1995 | 1995 | botocore.docs.params
import time: 1450 | 3445 | botocore.docs.method
import time: 1605 | 1605 | botocore.docs.sharedexample
import time: 2610 | 64372 | botocore.docs.client
import time: 1318 | 1318 | botocore.docs.paginator
import time: 1093 | 1093 | botocore.docs.waiter
import time: 3153 | 201481 | botocore.docs.service
import time: 1636 | 203117 | botocore.docs
import time: 1037 | 204154 | botocore.docs.docstring
import time: 2504 | 238930 | botocore.waiter
import time: 2959 | 2959 | botocore.eventstream
import time: 4886 | 7844 | botocore.parsers
import time: 2668 | 2668 | botocore.validate
import time: 3865 | 6532 | botocore.serialize
import time: 73 | 73 | _uuid
import time: 529 | 602 | uuid
import time: 979 | 979 | botocore.history
import time: 2781 | 2781 | botocore.hooks
import time: 1370 | 1370 | botocore.response
import time: 2575 | 3944 | botocore.httpchecksum
import time: 2545 | 10850 | botocore.endpoint
import time: 1332 | 12182 | botocore.config
import time: 5471 | 5471 | botocore.auth
import time: 957 | 957 | botocore.crt
import time: 3826 | 3826 | botocore.endpoint_provider
import time: 3396 | 13649 | botocore.regions
import time: 2541 | 2541 | botocore.signers
import time: 37 | 37 | botocore.customizations
import time: 26 | 63 | botocore.customizations.useragent
import time: 2052 | 2115 | botocore.useragent
import time: 2907 | 47767 | botocore.args
import time: 1283 | 1283 | botocore.compress
import time: 304 | 304 | termios
import time: 200 | 503 | getpass
import time: 484 | 484 | signal
import time: 276 | 276 | fcntl
import time: 89 | 89 | msvcrt
import time: 169 | 169 | _posixsubprocess
import time: 625 | 1641 | subprocess
import time: 1175 | 1175 | configparser
import time: 1404 | 2579 | botocore.configloader
import time: 2315 | 2315 | botocore.tokens
import time: 8087 | 15123 | botocore.credentials
import time: 4083 | 4083 | botocore.model
import time: 2047 | 6129 | botocore.discovery
import time: 3074 | 3074 | botocore.paginate
import time: 1002 | 1002 | botocore.retries
import time: 1167 | 1167 | botocore.retries.bucket
import time: 837 | 837 | botocore.retries.quota
import time: 612 | 612 | botocore.retries.base
import time: 882 | 1494 | botocore.retries.special
import time: 2382 | 4712 | botocore.retries.standard
import time: 873 | 873 | botocore.retries.throttling
import time: 2707 | 9457 | botocore.retries.adaptive
import time: 5685 | 328446 | botocore.client
import time: 1741 | 1741 | botocore.retryhandler
import time: 678 | 678 | botocore.translate
import time: 5960 | 8378 | botocore.handlers
import time: 2656 | 2656 | botocore.monitoring
import time: 3620 | 3620 | botocore.configprovider
import time: 1016 | 1016 | botocore.errorfactory
import time: 1885 | 1885 | botocore.loaders
import time: 5061 | 351634 | botocore.session
import time: 939 | 939 | boto3.utils
import time: 748 | 748 | boto3.resources
import time: 738 | 738 | boto3.docs.client
import time: 911 | 911 | boto3.docs.base
import time: 878 | 878 | boto3.docs.method
import time: 1310 | 1310 | boto3.docs.utils
import time: 1322 | 4420 | boto3.docs.action
import time: 812 | 812 | boto3.docs.attr
import time: 1542 | 1542 | boto3.docs.collection
import time: 1242 | 1242 | boto3.docs.subresource
import time: 1202 | 1202 | boto3.docs.waiter
import time: 2142 | 11359 | boto3.docs.resource
import time: 4560 | 16656 | boto3.docs.service
import time: 1104 | 17759 | boto3.docs
import time: 967 | 967 | boto3.docs.docstring
import time: 2357 | 2357 | boto3.resources.model
import time: 1125 | 1125 | boto3.resources.params
import time: 1329 | 1329 | boto3.resources.response
import time: 1261 | 6071 | boto3.resources.action
import time: 1101 | 1101 | boto3.resources.base
import time: 1884 | 1884 | boto3.resources.collection
import time: 4119 | 32645 | boto3.resources.factory
import time: 2010 | 387435 | boto3.session
import time: 3584 | 422564 | boto3
import time: 1792 | 1792 | numpy._utils._convertions
import time: 1065 | 2856 | numpy._utils
import time: 4987 | 7843 | numpy._globals
import time: 1247 | 1247 | numpy.exceptions
import time: 812 | 812 | numpy.version
import time: 37 | 37 | numpy._distributor_init_local
import time: 1031 | 1067 | numpy._distributor_init
import time: 1391 | 1391 | numpy._utils._inspect
import time: 4203 | 4203 | numpy.core._exceptions
import time: 2220 | 2220 | numpy.dtypes
import time: 47163 | 53585 | numpy.core._multiarray_umath
import time: 2926 | 57902 | numpy.core.overrides
import time: 7971 | 65872 | numpy.core.multiarray
import time: 5783 | 5783 | numpy.core.umath
import time: 2581 | 2581 | numpy.core._string_helpers
import time: 213 | 213 | pickle5
import time: 577 | 577 | _compat_pickle
import time: 977 | 977 | _pickle
import time: 2721 | 4274 | pickle
import time: 4231 | 8717 | numpy.compat.py3k
import time: 3481 | 12197 | numpy.compat
import time: 5065 | 5065 | numpy.core._dtype
import time: 6900 | 24161 | numpy.core._type_aliases
import time: 8037 | 34778 | numpy.core.numerictypes
import time: 469 | 469 | _contextvars
import time: 689 | 1157 | contextvars
import time: 3892 | 5048 | numpy.core._ufunc_config
import time: 7476 | 12524 | numpy.core._methods
import time: 14225 | 26748 | numpy.core.fromnumeric
import time: 7659 | 34407 | numpy.core.shape_base
import time: 18204 | 18204 | numpy.core.arrayprint
import time: 3078 | 3078 | numpy.core._asarray
import time: 17565 | 73253 | numpy.core.numeric
import time: 17894 | 17894 | numpy.core.defchararray
import time: 10865 | 10865 | numpy.core.records
import time: 8270 | 8270 | numpy.core.memmap
import time: 8022 | 8022 | numpy.core.function_base
import time: 10691 | 10691 | numpy.core._machar
import time: 12212 | 12212 | numpy.core.getlimits
import time: 18510 | 18510 | numpy.core.einsumfunc
import time: 4183 | 4183 | numpy.core._multiarray_tests
import time: 19182 | 23365 | numpy.core._add_newdocs
import time: 2135 | 2135 | numpy.core._add_newdocs_scalars
import time: 1353 | 1353 | numpy.core._dtype_ctypes
import time: 512 | 512 | _ctypes
import time: 290 | 290 | ctypes._endian
import time: 2091 | 2892 | ctypes
import time: 4432 | 7324 | numpy.core._internal
import time: 1356 | 1356 | numpy._pytesttester
import time: 8381 | 310056 | numpy.core
import time: 34 | 310089 | numpy.core._multiarray_umath
import time: 1585 | 311673 | numpy.__config__
import time: 6496 | 6496 | numpy.lib.mixins
import time: 1105 | 1105 | numpy.lib.ufunclike
import time: 2173 | 3278 | numpy.lib.type_check
import time: 3751 | 7028 | numpy.lib.scimath
import time: 1968 | 1968 | numpy.lib.stride_tricks
import time: 2291 | 4259 | numpy.lib.twodim_base
import time: 1709 | 1709 | numpy.linalg._umath_linalg
import time: 3938 | 3938 | numpy._typing._nested_sequence
import time: 668 | 668 | numpy._typing._nbit
import time: 1909 | 1909 | numpy._typing._char_codes
import time: 901 | 901 | numpy._typing._scalars
import time: 579 | 579 | numpy._typing._shape
import time: 2284 | 2284 | numpy._typing._dtype_like
import time: 2896 | 2896 | numpy._typing._array_like
import time: 1954 | 15125 | numpy._typing
import time: 8540 | 29631 | numpy.linalg.linalg
import time: 1296 | 30926 | numpy.linalg
import time: 4079 | 35005 | numpy.matrixlib.defmatrix
import time: 1447 | 36451 | numpy.matrixlib
import time: 3048 | 3048 | numpy.lib.histograms
import time: 13808 | 16856 | numpy.lib.function_base
import time: 3018 | 56324 | numpy.lib.index_tricks
import time: 3627 | 3627 | numpy.lib.nanfunctions
import time: 2984 | 2984 | numpy.lib.shape_base
import time: 4077 | 4077 | numpy.lib.polynomial
import time: 4251 | 4251 | numpy.lib.utils
import time: 2735 | 2735 | numpy.lib.arraysetops
import time: 2799 | 2799 | numpy.lib.format
import time: 2171 | 2171 | numpy.lib._datasource
import time: 3395 | 3395 | numpy.lib._iotools
import time: 7182 | 15546 | numpy.lib.npyio
import time: 1428 | 1428 | numpy.lib.arrayterator
import time: 2928 | 2928 | numpy.lib.arraypad
import time: 1181 | 1181 | numpy.lib._version
import time: 2177 | 110776 | numpy.lib
import time: 943 | 943 | numpy.fft._pocketfft_internal
import time: 3747 | 4690 | numpy.fft._pocketfft
import time: 1083 | 1083 | numpy.fft.helper
import time: 1119 | 6891 | numpy.fft
import time: 2703 | 2703 | numpy.polynomial.polyutils
import time: 3927 | 3927 | numpy.polynomial._polybase
import time: 8857 | 15487 | numpy.polynomial.polynomial
import time: 3503 | 3503 | numpy.polynomial.chebyshev
import time: 2790 | 2790 | numpy.polynomial.legendre
import time: 2856 | 2856 | numpy.polynomial.hermite
import time: 2929 | 2929 | numpy.polynomial.hermite_e
import time: 2790 | 2790 | numpy.polynomial.laguerre
import time: 2001 | 32353 | numpy.polynomial
import time: 78 | 78 | backports_abc
import time: 4047 | 4124 | numpy.random._common
import time: 172 | 172 | secrets
import time: 2772 | 7067 | numpy.random.bit_generator
import time: 1662 | 1662 | numpy.random._bounded_integers
import time: 1472 | 1472 | numpy.random._mt19937
import time: 8317 | 18517 | numpy.random.mtrand
import time: 1542 | 1542 | numpy.random._philox
import time: 1664 | 1664 | numpy.random._pcg64
import time: 1384 | 1384 | numpy.random._sfc64
import time: 3873 | 3873 | numpy.random._generator
import time: 4611 | 31589 | numpy.random._pickle
import time: 1380 | 32968 | numpy.random
import time: 2748 | 2748 | numpy.ctypeslib
import time: 29009 | 29009 | numpy.ma.core
import time: 6710 | 6710 | numpy.ma.extras
import time: 1133 | 36852 | numpy.ma
import time: 13048 | 558273 | numpy
import time: 89 | 89 | chardet
import time: 4947 | 4947 | charset_normalizer.constant
import time: 1019 | 1019 | charset_normalizer.md__mypyc
import time: 292 | 292 | unicodedata
import time: 240 | 240 | _multibytecodec
import time: 2500 | 3031 | charset_normalizer.utils
import time: 2105 | 6154 | charset_normalizer.md
import time: 3039 | 3039 | charset_normalizer.models
import time: 2431 | 16570 | charset_normalizer.cd
import time: 4711 | 21280 | charset_normalizer.api
import time: 992 | 992 | charset_normalizer.legacy
import time: 717 | 717 | charset_normalizer.version
import time: 1189 | 24176 | charset_normalizer
import time: 2043 | 2043 | http.cookiejar
import time: 1044 | 1044 | http.cookies
import time: 964 | 28314 | requests.compat
import time: 6070 | 34383 | requests.exceptions
import time: 110 | 110 | chardet
import time: 70 | 70 | chardet
import time: 2833 | 2833 | idna.package_data
import time: 3689 | 3689 | idna.idnadata
import time: 905 | 905 | idna.intranges
import time: 3078 | 7671 | idna.core
import time: 1230 | 11732 | idna
import time: 1293 | 13094 | requests.packages
import time: 96 | 96 | zipfile._path.glob
import time: 401 | 497 | zipfile._path
import time: 139 | 139 | zipfile.__main__
import time: 651 | 1285 | zipfile
import time: 670 | 670 | requests.certs
import time: 904 | 904 | requests.__version__
import time: 991 | 991 | requests._internal_utils
import time: 2560 | 2560 | requests.cookies
import time: 1027 | 1027 | requests.structures
import time: 91 | 91 | importlib.resources._itertools
import time: 287 | 378 | importlib.resources.readers
import time: 144 | 522 | importlib.readers
import time: 4215 | 12171 | requests.utils
import time: 1928 | 1928 | requests.auth
import time: 256 | 256 | stringprep
import time: 277 | 533 | encodings.idna
import time: 750 | 750 | requests.hooks
import time: 1594 | 1594 | requests.status_codes
import time: 3883 | 6759 | requests.models
import time: 80 | 80 | socks
import time: 1465 | 1545 | urllib3.contrib.socks
import time: 2259 | 12489 | requests.adapters
import time: 2922 | 15411 | requests.sessions
import time: 940 | 16351 | requests.api
import time: 2190 | 78298 | requests
2. AWS Lambda 関数 (ZIP) x Layer x -X importtime
オプション
AWS Lambda では拡張機能 (Lambda extensions) を使ってランタイムに任意のオプションを追加することができる.以下のドキュメントにはラッパースクリプトを書いて Python ランタイムに -X importtime
オプションを追加する例が載っている👀
docs.aws.amazon.com
👾 template.yaml
AWS SAM の template.yaml
を以下のように書いた.
Python ランタイムに -X importtime
オプションを追加するラッパースクリプトを含んだ AWS Lambda Layer を作る.そして AWS Lambda 関数の環境変数 AWS_LAMBDA_EXEC_WRAPPER
に /opt/importtime_wrapper
を設定する.
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Resources:
Function:
Type: AWS::Serverless::Function
Properties:
FunctionName: importtime-wrapper
CodeUri: src/
Handler: app.lambda_handler
Runtime: python3.12
Architectures:
- x86_64
Layers:
- !Ref Layer
Environment:
Variables:
AWS_LAMBDA_EXEC_WRAPPER: /opt/importtime_wrapper
Layer:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: importtime-wrapper-layer
ContentUri: layer/src/
CompatibleRuntimes:
- python3.12
👾 importtime_wrapper
importtime_wrapper
はドキュメントの通りにしておく.そして,デプロイする前に忘れずに chmod +x layer/src/importtime_wrapper
コマンドを実行してラッパースクリプトに実行権限を与えておく.
args=("$@")
extra_args=("-X" "importtime")
args=("${args[@]:0:$#-1}" "${extra_args[@]}" "${args[@]: -1}")
exec "${args[@]}"
実行結果(割愛)
AWS Lambda 関数 importtime-wrapper
をデプロイして実行すると同じく import 時間がログに出力されていた❗️
3. AWS Lambda 関数 (Image) x PYTHONPROFILEIMPORTTIME
環境変数
👾 template.yaml
AWS SAM の template.yaml
を以下のように書いた.
Environment.Variables
で PYTHONPROFILEIMPORTTIME: 1
を設定している😃
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Resources:
Function:
Type: AWS::Serverless::Function
Properties:
FunctionName: importtime-env-image
PackageType: Image
Architectures:
- x86_64
Environment:
Variables:
PYTHONPROFILEIMPORTTIME: 1
Metadata:
Dockerfile: Dockerfile
DockerContext: ./src
DockerTag: python3.12-v1
👾 Dockerfile
FROM public.ecr.aws/lambda/python:3.12
COPY app.py requirements.txt ./
RUN python3.12 -m pip install -r requirements.txt -t .
CMD ["app.lambda_handler"]
実行結果(割愛)
AWS Lambda 関数 importtime-env-image
をデプロイして実行すると同じく import 時間がログに出力されていた❗️
4. AWS Lambda 関数 (Image) x -X importtime
オプション
以下のドキュメントを参考に python:3.12
など一般的なコンテナイメージ (non-AWS base image) に AWS Lambda Runtime Interface Client (RIC) をインストールして実行するパターンを試す.Dockerfile のサンプルもドキュメントを参考にした.
docs.aws.amazon.com
👾 template.yaml
AWS SAM の template.yaml
を以下のように書いた.
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Resources:
Function:
Type: AWS::Serverless::Function
Properties:
FunctionName: importtime-option-image
PackageType: Image
Architectures:
- x86_64
Metadata:
Dockerfile: Dockerfile
DockerContext: ./src
DockerTag: python3.12-v1
👾 Dockerfile
ポイントは ENTRYPOINT
に -X importtime
オプションを追加しているところ📝
ARG FUNCTION_DIR="/function"
FROM python:3.12 as build-image
ARG FUNCTION_DIR
RUN mkdir -p ${FUNCTION_DIR}
COPY . ${FUNCTION_DIR}
RUN pip install --target ${FUNCTION_DIR} -r ${FUNCTION_DIR}/requirements.txt
FROM python:3.12-slim
ARG FUNCTION_DIR
WORKDIR ${FUNCTION_DIR}
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ENTRYPOINT [ "/usr/local/bin/python", "-X", "importtime", "-m", "awslambdaric" ]
CMD ["app.lambda_handler"]
👾 requirements.txt
requirements.txt
に awslambdaric
を追加した.
awslambdaric
boto3
numpy
requests
実行結果(割愛)
AWS Lambda 関数 importtime-option-image
をデプロイして実行すると同じく import 時間がログに出力されていた❗️
まとめ
AWS Lambda 関数 (Python) で import の最適化をするときには Python の -X importtime
オプション(もしくは環境変数 PYTHONPROFILEIMPORTTIME
)が便利〜👏 という紹介記事でした \( 'ω')/
参考記事
以下の記事でも AWS Lambda 関数 (Python) に PYTHONPROFILEIMPORTTIME
環境変数を設定して INIT を最適化する方法が紹介されているので合わせて読んでみると良いかも📝
aws.amazon.com